Temporal element restoration in augmented reality systems

ABSTRACT

Methods, apparatuses, computer program products, devices and systems are described that carry out accepting a request associated with at least one of an item, an aspect, or an element that is not present in a field of view of a user&#39;s augmented reality device; presenting in a display of the augmented reality device at least one augmented reality representation related to the at least one item, aspect, or element in response to accepting a request associated with at least one item, aspect, or element that is not present in a field of view of an augmented reality device; and processing the request and any related interaction of the user via the at least one augmented reality representation.

If an Application Data Sheet (ADS) has been filed on the filing date ofthis application, it is incorporated by reference herein. Anyapplications claimed on the ADS for priority under 35 U.S.C. §§ 119,120, 121, or 365(c), and any and all parent, grandparent,great-grandparent, etc. applications of such applications, are alsoincorporated by reference, including any priority claims made in thoseapplications and any material incorporated by reference, to the extentsuch subject matter is not inconsistent herewith.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is related to and/or claims the benefit of theearliest available effective filing date(s) from the following listedapplication(s) (the “Priority Applications”), if any, listed below(e.g., claims earliest available priority dates for other thanprovisional patent applications or claims benefits under 35 USC § 119(e)for provisional patent applications, for any and all parent,grandparent, greatgrandparent, etc. applications of the PriorityApplication(s)). In addition, the present application is related to the“Related Applications,” if any, listed below.

Priority Applications

NONE

Related Applications

U.S. patent application Ser. No. 13/646,147, entitled FORMATTING OF ONEOR MORE PERSISTENT AUGMENTATIONS IN AN AUGMENTED VIEW IN RESPONSE TOMULTIPLE INPUT FACTORS, naming Gene Fein, Royce A. Levien, Richard T.Lord, Robert W. Lord, Mark A. Malamud, John D. Rinaldo, Jr., andClarence T. Tegreene as inventors, filed 5 Oct. 2012 with attorneydocket no. 0412-003-001-000000, is related to the present application.

U.S. patent application Ser. No. 13/648,012, entitled FORMATTING OF ONEOR MORE PERSISTENT AUGMENTATIONS IN AN AUGMENTED VIEW IN RESPONSE TOMULTIPLE INPUT FACTORS, naming Gene Fein, Royce A. Levien, Richard T.Lord, Robert W. Lord, Mark A. Malamud, John D. Rinaldo, Jr., andClarence T. Tegreene as inventors, filed 9 Oct. 2012 with attorneydocket no. 0412-003-001-000001, is related to the present application.

U.S. patent application Ser. No. 13/672,575, entitled PRESENTING ANAUGMENTED VIEW IN RESPONSE TO ACQUISITION OF DATA INFERRING USERACTIVITY, naming Gene Fein, Royce A. Levien, Richard T. Lord, Robert W.Lord, Mark A. Malamud, John D. Rinaldo, Jr., and Clarence T. Tegreene asinventors, filed 9 Nov. 2012 with attorney docket no.0412-003-002-000000, is related to the present application.

U.S. patent application Ser. No. 13/673,070, entitled PRESENTING ANAUGMENTED VIEW IN RESPONSE TO ACQUISITION OF DATA INFERRING USERACTIVITY, naming Gene Fein, Royce A. Levien, Richard T. Lord, Robert W.Lord, Mark A. Malamud, John D. Rinaldo, Jr., and Clarence T. Tegreene asinventors, filed 9 Nov. 2012 with attorney docket no.0412-003-002-000001, is related to the present application.

U.S. patent application Ser. No. 13/689,372, entitled SYSTEMS ANDMETHODS FOR SHARING AUGMENTATION DATA, naming Gene Fein, Royce A.Levien, Richard T. Lord, Robert W. Lord, Mark A. Malamud, John D.Rinaldo, Jr., and Clarence T. Tegreene as inventors, filed 29 Nov. 2012with attorney docket no. 0412-003-003-000000, is related to the presentapplication.

U.S. patent application Ser. No. 13/690,003, entitled SYSTEMS ANDMETHODS FOR SHARING AUGMENTATION DATA, naming Gene Fein, Royce A.Levien, Richard T. Lord, Robert W. Lord, Mark A. Malamud, John D.Rinaldo, Jr., and Clarence T. Tegreene as inventors, filed 30 Nov. 2012with attorney docket no. 0412-003-003-000001, is related to the presentapplication.

U.S. patent application Ser. No. 13/709,465, entitled SYSTEMS ANDMETHODS FOR OBTAINING AND USING AUGMENTATION DATA AND FOR SHARING USAGEDATA, naming Gene Fein, Royce A. Levien, Richard T. Lord, Robert W.Lord, Mark A. Malamud, John D. Rinaldo, Jr., and Clarence T. Tegreene asinventors, filed 10 Dec. 2012 with attorney docket no.0412-003-004-000000, is related to the present application.

U.S. patent application Ser. No. 13/711,095, entitled SYSTEMS ANDMETHODS FOR OBTAINING AND USING AUGMENTATION DATA AND FOR SHARING USAGEDATA, naming Gene Fein, Royce A. Levien, Richard T. Lord, Robert W.Lord, Mark A. Malamud, John D. Rinaldo, Jr., and Clarence T. Tegreene asinventors, filed 11 Dec. 2012 with attorney docket no.0412-003-004-000001, is related to the present application.

U.S. patent application Ser. No. 13/721,340, entitled CORRELATING USERREACTION WITH AT LEAST AN ASPECT ASSOCIATED WITH AN AUGMENTATION OF ANAUGMENTED VIEW, naming Gene Fein, Royce A. Levien, Richard T. Lord,Robert W. Lord, Mark A. Malamud, John D. Rinaldo, Jr., and Clarence T.Tegreene as inventors, filed 20 Dec. 2012 with attorney docket no.0412-003-005-000000, is related to the present application.

U.S. patent application Ser. No. 13/723,610, entitled CORRELATING USERREACTION WITH AT LEAST AN ASPECT ASSOCIATED WITH AN AUGMENTATION OF ANAUGMENTED VIEW, naming Gene Fein, Royce A. Levien, Richard T. Lord,Robert W. Lord, Mark A. Malamud, John D. Rinaldo, Jr., and Clarence T.Tegreene as inventors, filed 21 Dec. 2012 with attorney docket no.0412-003-005-000001, is related to the present application.

U.S. patent application Ser. No. 13/729,278, entitled CORRELATING USERREACTIONS WITH AUGMENTATIONS DISPLAYED THROUGH AUGMENTED VIEWS, namingGene Fein, Royce A. Levien, Richard T. Lord, Robert W. Lord, Mark A.Malamud, John D. Rinaldo, Jr., and Clarence T. Tegreene as inventors,filed 28 Dec. 2012 with attorney docket no. 0412-003-006-000000, isrelated to the present application.

U.S. patent application Ser. No. 13/731,233, entitled CORRELATING USERREACTIONS WITH AUGMENTATIONS DISPLAYED THROUGH AUGMENTED VIEWS, namingGene Fein, Royce A. Levien, Richard T. Lord, Robert W. Lord, Mark A.Malamud, John D. Rinaldo, Jr., and Clarence T. Tegreene as inventors,filed 31 Dec. 2012 with attorney docket no. 0412-003-006-000001, isrelated to the present application.

U.S. patent application Ser. No. 13/768,048, entitled DISPLAYING INRESPONSE TO DETECTING ONE OR MORE USER BEHAVIORS ONE OR MORE SECONDAUGMENTATIONS THAT ARE BASED ON ONE OR MORE REGISTERED FIRSTAUGMENTATIONS, naming Gene Fein, Royce A. Levien, Richard T. Lord,Robert W. Lord, Mark A. Malamud, John D. Rinaldo, Jr., and Clarence T.Tegreene as inventors, filed 15 Feb. 2013 with attorney docket no.0412-003-007-000000, is related to the present application.

U.S. patent application Ser. No. 13/770,053, entitled DISPLAYING INRESPONSE TO DETECTING ONE OR MORE USER BEHAVIORS ONE OR MORE SECONDAUGMENTATIONS THAT ARE BASED ON ONE OR MORE REGISTERED FIRSTAUGMENTATIONS, naming Gene Fein, Royce A. Levien, Richard T. Lord,Robert W. Lord, Mark A. Malamud, John D. Rinaldo, Jr., and Clarence T.Tegreene as inventors, filed 19 Feb. 2013 with attorney docket no.0412-003-007-000001, is related to the present application.

The United States Patent Office (USPTO) has published a notice to theeffect that the USPTO's computer programs require that patent applicantsreference both a serial number and indicate whether an application is acontinuation, continuation-in-part, or divisional of a parentapplication. Stephen G. Kunin, Benefit of Prior-Filed Application, USPTOOfficial 5 Gazette Mar. 18, 2003. The USPTO further has provided formsfor the Application Data Sheet which allow automatic loading ofbibliographic data but which require identification of each applicationas a continuation, continuation-in-part, or divisional of a parentapplication. The present Applicant Entity (hereinafter “Applicant”) hasprovided above a specific reference to the application(s)from whichpriority is being claimed as recited by statute.

Applicant understands that the statute is unambiguous in its specificreference language and does not require either a serial number or anycharacterization, such as “continuation” or “continuation-in-part,” forclaiming priority to U.S. patent applications. Notwithstanding theforegoing, Applicant understands that the USPTO's computer programs havecertain data entry requirements, and hence Applicant has provideddesignation(s) of a relationship between the present application and itsparent application(s) as set forth above and in any ADS filed in thisapplication, but expressly points out that such designation(s) are notto be construed in any way as any type of commentary and/or admission asto whether or not the present application contains any new matter inaddition to the matter of its parent application(s).

If the listings of applications provided above are inconsistent with thelistings provided via an ADS, it is the intent of the Applicant to claimpriority to each application that appears in the Priority Applicationssection of the ADS and to each application that appears in the PriorityApplications section of this application.

All subject matter of the Priority Applications and the RelatedApplications and of any and all parent, grandparent, great-grandparent,etc. applications of the Priority Applications and the RelatedApplications, including any priority claims, is incorporated herein byreference to the extent such subject matter is not inconsistentherewith.

TECHNICAL FIELD

This description relates to data capture, data handling, and datadisplay techniques.

SUMMARY

An embodiment provides a system. In one implementation, the systemincludes but is not limited to circuitry for accepting a requestassociated with at least one of an item, an aspect, or an element thatis not present in a field of view of a user's augmented reality device;circuitry for presenting in a display of the augmented reality device atleast one augmented reality representation related to the at least oneitem, aspect, or element in response to at least one ouput of thecircuitry for accepting a request associated with at least one item,aspect, or element that is not present in a field of view of anaugmented reality device; and circuitry for processing the request andany related interaction of the user via the at least one augmentedreality representation. In addition to the foregoing, other systemaspects are described in the claims, drawings, and text forming a partof the present disclosure.

In one or more various aspects, related systems include but are notlimited to circuitry and/or programming for effecting theherein-referenced method aspects; the circuitry and/or programming canbe virtually any combination of hardware, software, and/or firmwareconfigured to effect the herein-referenced method aspects depending uponthe design choices of the system designer.

In one or more various aspects, related systems include but are notlimited to computing means and/or programming for effecting theherein-referenced method aspects; the computing means and/or programmingmay be virtually any combination of hardware, software, and/or firmwareconfigured to effect the herein-referenced method aspects depending uponthe design choices of the system designer.

An embodiment provides a computer-implemented method. In oneimplementation, the method includes but is not limited to accepting arequest associated with at least one of an item, an aspect, or anelement that is not present in a field of view of a user's augmentedreality device; presenting in a display of the augmented reality deviceat least one augmented reality representation related to the at leastone item, aspect, or element in response to accepting a requestassociated with at least one item, aspect, or element that is notpresent in a field of view of an augmented reality device; andprocessing the request and any related interaction of the user via theat least one augmented reality representation. In addition to theforegoing, other method aspects are described in the claims, drawings,and text forming a part of the present disclosure.

An embodiment provides an article of manufacture including a computerprogram product. In one implementation, the article of manufactureincludes but is not limited to a signal-bearing medium configured by oneor more instructions related to accepting a request associated with atleast one of an item, an aspect, or an element that is not present in afield of view of a user's augmented reality device; presenting in adisplay of the augmented reality device at least one augmented realityrepresentation related to the at least one item, aspect, or element inresponse to accepting a request associated with at least one item,aspect, or element that is not present in a field of view of anaugmented reality device; and processing the request and any relatedinteraction of the user via the at least one augmented realityrepresentation. In addition to the foregoing, other computer programproduct aspects are described in the claims, drawings, and text forminga part of the present disclosure.

An embodiment provides a system. In one implementation, the systemincludes but is not limited to a computing device and instructions. Theinstructions when executed on the computing device cause the computingdevice to accept a request associated with at least one of an item, anaspect, or an element that is not present in a field of view of a user'saugmented reality device; present in a display of the augmented realitydevice at least one augmented reality representation related to the atleast one item, aspect, or element in response to accepting a requestassociated with at least one item, aspect, or element that is notpresent in a field of view of an augmented reality device; and processthe request and any related interaction of the user via the at least oneaugmented reality representation. In addition to the foregoing, othersystem aspects are described in the claims, drawings, and text forming apart of the present disclosure.

In addition to the foregoing, various other method and/or system and/orprogram product aspects are set forth and described in the teachingssuch as text (e.g., claims and/or detailed description) and/or drawingsof the present disclosure.

The foregoing is a summary and thus may contain simplifications,generalizations, inclusions, and/or omissions of detail; consequently,those skilled in the art will appreciate that the summary isillustrative only and is NOT intended to be in any way limiting. Otheraspects, features, and advantages of the devices and/or processes and/orother subject matter described herein will become apparent in theteachings set forth herein.

BRIEF DESCRIPTION OF THE FIGURES

With reference now to FIG. 1, shown is a number of examples of augmentedreality devices.

FIG. 2 illustrates an augmented reality device and a real world field ofview of its camera.

FIG. 3 illustrates one embodiment wherein a user interacts with a systemto select, drag, and drop an augmented reality representation of a book.

FIGS. 4a, 4b, 4c, 4d, 4e, and 4f illustrate an example of a system forselecting, dragging, and dropping in augmented reality systems in whichembodiments may be implemented, perhaps in a device and/or through anetwork, which may serve as a context for introducing one or moreprocesses and/or devices described herein.

With reference now to FIG. 5, shown is an example of an operational flowrepresenting example operations related to selecting, dragging, anddropping in augmented reality systems, which may serve as a context forintroducing one or more processes and/or devices described herein.

FIG. 6 illustrates an alternative embodiment of the example operationalflow of FIG. 5.

FIG. 7 illustrates an alternative embodiment of the example operationalflow of FIG. 5.

FIG. 8 illustrates an alternative embodiment of the example operationalflow of FIG. 5.

FIG. 9 illustrates an alternative embodiment of the example operationalflow of FIG. 5.

FIG. 10 illustrates an alternative embodiment of the example operationalflow of FIG. 5.

FIG. 11 illustrates an alternative embodiment of the example operationalflow of FIG. 5.

FIG. 12 illustrates an alternative embodiment of the example operationalflow of FIG. 5.

With reference now to FIG. 13, shown is an example of an operationalflow representing example operations related to selecting, dragging, anddropping in augmented reality systems, which may serve as a context forintroducing one or more processes and/or devices described herein.

With reference now to FIG. 14, shown is an example of an operationalflow representing example operations related to selecting, dragging, anddropping in augmented reality systems, which may serve as a context forintroducing one or more processes and/or devices described herein.

FIG. 15 illustrates an example of a system for dynamically preservingscene elements in augmented reality systems in which embodiments may beimplemented, perhaps in a device and/or through a network, which mayserve as a context for introducing one or more processes and/or devicesdescribed herein.

FIGS. 16-18 depict a scenario in which there is no ability todynamically preserve scene elements in an augmented reality system.Shown is an example of a user attempting and failing to select adisplayed moving person.

FIGS. 19-23 depict a scenario in which there is an ability todynamically preserve scene elements in an augmented reality system.Shown is an example of a user attempting and succeeding to select andinteract with a displayed (initially) moving person.

With reference now to FIG. 24, shown is an example of an operationalflow representing example operations related to dynamically preservingscene elements in augmented reality systems, which may serve as acontext for introducing one or more processes and/or devices describedherein.

FIG. 25 illustrates an alternative embodiment of the example operationalflow of FIG. 24.

FIG. 26 illustrates an alternative embodiment of the example operationalflow of FIG. 24.

FIG. 27 illustrates an alternative embodiment of the example operationalflow of FIG. 24.

FIG. 28 illustrates an alternative embodiment of the example operationalflow of FIG. 24.

With reference now to FIG. 29, shown is an example of an operationalflow representing example operations related to dynamically preservingscene elements in augmented reality systems, which may serve as acontext for introducing one or more processes and/or devices describedherein.

FIG. 30 illustrates an alternative embodiment of the example operationalflow of FIG. 29.

FIG. 31 illustrates an example of a system for temporal elementrestoration in augmented reality systems in which embodiments may beimplemented, perhaps in a device and/or through a network, which mayserve as a context for introducing one or more processes and/or devicesdescribed herein.

FIGS. 32-40 depict stages in a scenario showing an example of temporalelement restoration in an augmented reality system. Shown are stages ofa user reserving a taxi cab seen through an augmented reality device,and later confirming the reservation through interaction with anaugmented reality representation of the taxi cab superimposed on a scenein which the taxi cab is not actually present.

With reference now to FIG. 41, shown is an example of an operationalflow representing example operations related to temporal elementrestoration in an augmented reality system, which may serve as a contextfor introducing one or more processes and/or devices described herein.

FIG. 42 illustrates an alternative embodiment of the example operationalflow of FIG. 41.

FIG. 43 illustrates an alternative embodiment of the example operationalflow of FIG. 41.

FIG. 44 illustrates an alternative embodiment of the example operationalflow of FIG. 41.

FIG. 45 illustrates an alternative embodiment of the example operationalflow of FIG. 41.

FIG. 46 illustrates an alternative embodiment of the example operationalflow of FIG. 41.

FIG. 47 illustrates an alternative embodiment of the example operationalflow of FIG. 41.

The use of the same symbols in different drawings typically indicatessimilar or identical items unless context dictates otherwise.

DETAILED DESCRIPTION

In a world where people interact through augmented reality devices, suchdedicated augmented reality devices (e.g., Google Glass eyeglasses),smartphones, digital cameras, camcorders, and tablets, the augmentedreality display or interface provides a window on the real world, ontowhich is layered one or more computer generated objects, digital images,or functions. Stucturally and semantically, an augmented reality userinterface is fundamentally responsive to a physical state in physicalproximity to the user's device. Aspects of physical reality aretypically represented on the screen; however even if they are notrepresented on the screen they typically still affect what's happeningon the screen to some extent. This may be contrasted with virtualreality, in which a user's senses are typically fed a completelycomputer generated theme or environment, as an artificial sensorium.

Cross-Reality Drag and Drop

As a courtesy to the reader, and with reference to the accompanyingfigures herein, in general “100 series” reference numerals willtypically refer to items first introduced/described by FIG. 1, “200series” reference numerals will typically refer to items firstintroduced/described by FIG. 2, “300 series” reference numerals willtypically refer to items first introduced/described by FIG. 3, etc.

In the following detailed description, reference is made to theaccompanying drawings, which form a part hereof. In the drawings,similar symbols typically identify similar components, unless contextdictates otherwise. The illustrative embodiments described in thedetailed description, drawings, and claims are not meant to be limiting.Other embodiments may be utilized, and other changes may be made,without departing from the spirit or scope of the subject matterpresented here.

By way of background, the traditional computer screen “desktop” areaincludes a drag-and-drop functionality that allows for powerfulmanipulation of graphical objects and environments. This typicallyinvolves 1) a source, 2) an object, and 3) a destination. These threethings may determine the semantics of operations in the drag-and-dropprocess.

In an augmented reality situation, as described herein, a user mayperform a drag-and-drop operation from the real world into an augmentedreality (“AR”) view or display, and vice versa. For example, if a useris wearing AR glasses in a bookstore, there may be an AR shopping cartdisplayed on the glasses, viewable to the user. The user may then find areal book on a book shelf at the book store, point to the real book,peel off or otherwise pull an augmented reality representation of thebook into the AR shopping cart. When the user arrives at a checkoutstand or register to buy the book, the user may grab the AR book fromthe shopping cart, drag it onto the real cash register at which pointpayment may be initiated by the bookstore, and the transactioncompleted. The user may also select options for delivery of a physicalbook to herself or as a gift to someone else, and/or delivery of anelectronic book to a device.

In another example, a user sitting at home in their living room may viewher AR device on which is displayed an augmented reality representationof stacks of DVD's, which are functional links to Netflix videos, forexample. The user may reach out and grab an augmented realityrepresentation of a video, Star Wars, for example, and drag it onto thetelevision in the living room, thereby signaling to Netflix to beginplay of a streaming video of Star Wars on the (internet-ready)television, with concomitant notation in the user's Netflix account asto what was watched by the user when, and on which device. In somescenarios this may involve an associated debit to a credit account orbank account.

In another example, a user in the lobby of a movie theater may see amovie poster for the latest installment in the Star Wars saga, due outin theaters next year. The user may grab an an augmented realityrepresentation of the movie poster to an augmented reality wishlist onher augmented reality display, thereby updating her Netflix queue, forexample, to schedule notification when the movie is being shown at thetheater, and/or when it is available on Netflix for viewing.

In each of these examples, a camera or other detector will recognize andmark the source of an action, in other words the start of the “drag.”This is the object to be dragged. The camera or other detector will thenmonitor the “drag” or the motion away from the source object, andfinally the camera or other detector will recognize or mark thedestination, or “drop.” This is the place of significance where theaugmented reality representation is going. Each end of the action may bemarked explicitly by the user, for example by voice, touch (of theobject or of the AR device), gesture, or other signal.

Unlike traditional drag-and-drop on a computer desktop environment, notonly is there a recognition step, but the user is pointing to somethingin a scene in which there are a limited number of targets available,which serves to constrain the recognition problem. In one embodiment, aconstraint may be that a movie player app, like hulu or Netflix isrunning on the AR device or on another device like a television inproximity to the user. In another example, if an e-reader such as akindle device is open during a book-shopping experience, that may beused as a constraint to tell the system to look for books in theenvironment during the recognition step.

Recognition of the intended object(s) will typically happen via imagedata from a camera viewing a scene through an AR device. Context thatthe user is in may be taken into account. For example, the AR device mayrecognize types of stores or collections of items such as books orDVD's; or even diverse collections of objects such as items at a grocerystore.

Voice may be used to signal the correct recognition for “grabbing” anobject prior to dragging it. Other ways of marking the start of the dragmay also be used such as touching the object, tapping the object,touching a sensitive part of the AR device itself such as a button ortouchscreen, and/or making a gesture that was pre-programmed in to theAR device to tell the system that a selection had been made fordragging.

In one embodiment, speech alone may be used to drag-and-drop anaugmented reality representation.

In another embodiment, eye tracking may be used to identify, recognize,and select what the user is looking at, track the arc of movement,dragging, or transfer, and identify, recognize, and select thedestination for the drop.

As used herein, “augmentation”, “virtual” or “augmented realityrepresentation” may refer to things that are added to a display of areal scene, such as computer generated images, text, or photographs.

In one embodiment, a system might include a hand-held augmented realitydevice, with at least one sensor (such as a camera), at least onegraphical display for user output, and at least one touch screen (orother similar device) for user input. As directed by the user, theaugmented reality device may activate and display an augmented realityscene that includes real interface objects (such as those imaged by theaugmented reality device's camera) and at least one augmented realityrepresentation of an object.

In one embodiment, a real interface object in an augmented realitydisplay is detected and selected (e.g., by a first gesture, voicecommand, touch, or some other predetermined method) and then moved(e.g., the augmented reality device tracks movement using a secondgesture, voice command, or some other predetermined method) within theaugmented reality interface as an augmented reality (or virtual)representation of that object, either leaving the first real interfaceobject unmodified, or removing the first real interface object from thescene. In response to selection and movement of the real interfaceobject in the augmented reality interface, at least one destination fordropping the augmented reality representation of the object is presentedin the augmented reality interface, perhaps in proximity to the realinterface object. Destinations on the display for a drop may include athumbnail photo, an icon, or some other symbol that in some cases willconvey functionality upon the augmented reality representation of theobject when dropped. A destination icon or symbol represents a targetupon which the representation of the real interface object may bedropped (e.g., by a third gesture, voice command, or some otherpredetermined method).

For example, imagine a user is viewing an augmented reality scene in aretail store. She will see the real objects in the store (such as books,microwave ovens, and housewares) as well as virtual objects in theaugmented reality display (such as product annotations and a shoppingcart that follows her wherever she goes). If she wishes to purchase oneof the books she sees on a shelf, within the augmented reality interfaceshe may “pick up” a representation of all twelve volumes of the realOxford English Dictionary with a hand gesture, drag, and then drop theaugmented reality representation of them into her virtual shopping cartfor check-out, at which point she may decide, for example, to purchaseeither a real or an electronic copy of the book, or both.

In another embodiment, a virtual interface object in an augmentedreality display is selected (by a first gesture, voice command, touch,or some other pre-determined method) and then moved (by a secondgesture, voice command, or some other predetermined method) within theaugmented reality interface. In response to selection and movement ofthe virtual interface object in the augmented reality interface, atleast one real interface object may be presented in the augmentedreality interface in proximity of the virtual interface object. Eachreal interface object in the augmented reality interface represents atarget upon which the virtual interface object may be dropped (by athird gesture, voice command, or some other predetermined method).

For example, imagine you are viewing an augmented reality scene of yourfamily rec room. You see all the real objects in the room (such as atelevision, a table, a couch, bookshelves, et cetera) overlaid withaugmentations (such as the list of digital movies you own, perhapsrepresented by a stack of virtual DVDs on the table by the TV). You wishto watch one of the digital James Bond movies you own, so within theaugmented reality interface you pick up the virtual Goldfinger DVD,drag, and drop it on the real television screen. The movie then beginsto play on the real television (or it might be overlaid as anaugmentation to the real television so only the user sees it, or both).

In another example, a user is given a photograph by a friend, and wouldlike to post it on her social network page, such as her Facebook page.She may select the photograph with a hand gesture or voice, drag theresulting augmented reality representation of the photograph to a Fbicon in the corner of her augmented reality display, and drop it thereto register her Facebook page as a destination for a digital copy ofphotograph to go. This works similarly for images to be added toPinterest, notes to be added to a personal electronic diary, and otherpersonal data repositories.

As a courtesy to the reader, and with reference to the accompanyingfigures herein, in general “100 series” reference numerals willtypically refer to items first introduced/described by FIG. 1, “200series” reference numerals will typically refer to items firstintroduced/described by FIG. 2, “300 series” reference numerals willtypically refer to items first introduced/described by FIG. 3, etc.

In the following detailed description, reference is made to theaccompanying drawings, which form a part hereof. In the drawings,similar symbols typically identify similar components, unless contextdictates otherwise. The illustrative embodiments described in thedetailed description, drawings, and claims are not meant to be limiting.Other embodiments may be utilized, and other changes may be made,without departing from the spirit or scope of the subject matterpresented here.

FIG. 1 shows several devices that may be used for augmented realityinteractions with a user. These include tablet device 100 having tabletcamera screen 102, smartphone 104 having smartphone camera screen 106,digital camera 108, augmented reality eyeglasses 110 (showing anaugmentation in the form of compass heading, e.g., “SW,” and ambienttemperature, e.g., “65° F.”), and video camera 112. Other form factorsmay be fabricated having the functionality described herein.

FIG. 2 shows an augmented reality device (smartphone) 204 having anaugmented reality display 208, which depicts an image of the real worldfield of view of the augmented reality device (smartphone camera fieldof view) 200, including the augmentated reality representation 206,e.g., “SW 65° F.”

FIG. 3 illustrates an example augmented reality system 322 in whichembodiments may be implemented. The system 322 may work through anaugmented reality device 302 for use by user 300. Augmented realitysystem 322 may be implemented on augmented reality device 302, or it maybe implemented remotely, in whole or in part, for example as a cloudservice through network 304 to communicate with augmented reality device302. The augmented reality system 322 may contain, for example,environmental context evaluation module 306, augmented reality devicecontext evaluation module 308, object selection module 310, imageprocessing module 312, image database 314, digital image generationmodule 316, user movement tracking module 318, destination selectionmodule 319, and drop registration module 320. Augmented reality system322 running on or through augmented reality device 302 may communicateover a network 304, wirelessly or by hardwired connection. By way ofnetwork 304, which may include a cloud computing component, augmentedreality system 322 may communicate with network payment system 324,including credit account 326, google wallet 328, and/or PayPal 330.Augmented reality system 322 may also communicate via network 304 with aretailer 332, such as Target 334. Augmented reality system 322 may alsocommunicate via network 304 with online data service 336 such asFacebook 338, iTunes 340, and/or Google Play app store 342.

In this way, a user may interact with digital representations of herenvironment in order to, inter alia, complete transactions, collectitems of interest, e.g., digital media including digital images of realobjects, or manipulate things such as movies and games for viewing orplay, respectively.

As referenced herein, the augmented reality system 322 may be used toperform various data querying and/or recall techniques with respect toreal world objects and/or augmented reality representations of realworld objects. For example, where real world object image data isorganized, keyed to, and/or otherwise accessible using one or more imagedatabases, augmented reality system 322 may employ various Boolean,statistical, and/or semi-boolean searching techniques to select acorrect real world object image among a set of images in a real worldscene, e.g., by object selection module 310, and also to provide anaugmented reality representation of the object, either by finding onein, e.g., image database 314, or by generating one, e.g., via digitalimage generation module 316.

Many examples of databases and database structures may be used inconnection with augmented reality system 322. Such examples includehierarchical models (in which data is organized in a tree and/orparent-child node structure), network models (based on set theory, andin which multi-parent structures per child node are supported), orobject/relational models (combining the relational model with theobject-oriented model).

Still other examples include various types of eXtensible Mark-upLanguage (XML) databases. For example, a database may be included thatholds data in some format other than XML, but that is associated with anXML interface for accessing the database using XML. As another example,a database may store XML data directly. Additionally, or alternatively,virtually any semi-structured database may be used, so that context maybe provided to/associated with stored data elements (either encoded withthe data elements, or encoded externally to the data elements), so thatdata storage and/or access may be facilitated.

Such databases, and/or other memory storage techniques, may be writtenand/or implemented using various programming or coding languages. Forexample, object-oriented database management systems may be written inprogramming languages such as, for example, C++ or Java. Relationaland/or object/relational models may make use of database languages, suchas, for example, the structured query language (SQL), which may be used,for example, for interactive queries for disambiguating informationand/or for gathering and/or compiling data from the relationaldatabase(s).

For example, SQL or SQL-like operations over one or more real worldobject image data may be performed, or Boolean operations using realworld object image data 301 may be performed. For example, weightedBoolean operations may be performed in which different weights orpriorities are assigned to one or more real world object imagesdepending on context of the scene or context of the device 302,including other programs running on the device 302, perhaps relative toone another. For example, a number-weighted, exclusive-OR operation maybe performed to request specific weightings of categories of objectsdepending on recognized cues such as geodata indicating location at abookstore, e.g.

FIG. 4 illustrates an example of a user interaction with the instantaugmented reality system. FIG. 4a depicts an augmented reality device (asmartphone showing on its display a bookshelf containing books in thecamera's field of view.

FIG. 4b depicts a user's hand pointing to a book on one of the shelves;this gesture is detected by e.g., augmented reality system 322 and/orimage processing module 312, which may capture text printed on the spineof the book in the vicinity of, or touched by, the user's index finger.Further, augmented reality device context evaluation module 308 maydetect that the device is running a program having virtual shopping cartfunctionality associated with a specific bookstore (as indicated by theshopping cart image in the lower left corner of FIGS. 4b-4f ), and, ifthere were other non-book items in the scene, the system may use thebookstore-related virtual shopping cart as a filter such that only booksin the scene are considered for selection. In some embodiments, a menu,e.g., a drop-down menu of book titles for example may be presented forthe user to select from.

Upon selection of a book, augmented reality system 322 and/or digitalimage generation module 316 may find in image database 314 and display,or create and display an augmented reality representation 417 of theselected book.

FIG. 4c depicts that a single book on the bookshelf is highlighted,corresponding to the one pointed to by the user's index finger.

FIG. 4d depicts a more detailed augmented reality representation 417 ofthe selected book in association with the user's hand as the hand movestoward the shopping cart icon on the display. This is the moving ordragging operation that will, upon the book's arrival at the shoppingcart, tell the system that the information about the book should berecorded in the user's shopping cart account, perhaps on the bookstore'swebpage. This is registration of the drop. For example, destinationselection module 319 and/or drop registration module 320 may registerthe displayed augmented reality representation of the book at theshopping cart icon in the display of the augmented reality device inresponse to detecting that the user moved his pointing hand to the icon,as tracked by, e.g., user movement tracking module 318.

Optionally, the augmented reality display may provide an indication ofregistration of the drop, as shown in FIG. 4f , in which the shoppingcart icon has been modified to include a 1 on it, indicating that thereis one item in the cart.

The converse operations may also be carried out by augmented realitysystem 322, from AR to reality. This includes detecting an augmentedreality representation 417 on a display, moving the displayed augmentedreality representation 417 on the display of the augmented realitydevice according to at least one detected second action of the user(e.g., dragging it onto a real world item), and registering thedisplayed augmented reality representation at a location in a real worldfield of view of the augmented reality device in response to e.g., adragging gesture ending at a credit card processing device for payingfor a book, ending at a television for playing a movie, or ending at acar for transferring an audiobook to the car from a smartphone, forexample.

And of course a reality-to-AR and back again process can be performed bythe system, as shown in FIG. 14. One example of this is the completeprocess of detecting/selecting a real item indicated by a user, draggingan augmented reality representation 417 of it to a place on an ARdevice, then detecting/selecting it again for movement to a differentreal world object. One example of this is the complete process ofchoosing a book from a bookshelf at a bookstore, placing it in a virtualshopping cart, and then retrieving it for payment at a credit cardprocessing device.

FIGS. 5-14 illustrate operational flows representing example operationsrelated to selecting, dragging, and dropping in augmented realitysystems. In the following figures that include various examples ofoperational flows, discussion and explanation may be provided withrespect to the above-described system environments of FIGS. 1-4, and/orwith respect to other examples and contexts. However, it should beunderstood that the operational flows may be executed in a number ofother environments and contexts and/or in modified versions of FIGS.1-4. Also, although the various operational flows are presented in thesequence(s) illustrated, it should be understood that the variousoperations may be performed in other orders than those which areillustrated, or may be performed concurrently.

Dynamically Preserving Scene Elements in Augmented Reality Systems

In a situation in which a user is looking at a real world scene throughAR eyeglasses, for example, the user may want to select some object orperson within the scene to interact with via the AR glasses. Forexample, if a user sees David Bowie throught his glasses, she may wantto select the image of David Bowie seen through the AR glasses toactivate an option to buy some of David Bowie's music online, perhapsfor wireless download to the AR glasses. User input may includeeyetracking, voice, gesture, or touch on the AR device or on anotherdevice such as a smartphone linked to the AR glasses (e.g., viaBluetooth), among other input modalities.

The present application, in one embodiment, provides a system in whichthe elements of a scene presented on an AR device may be modified ortransformed in ways that preserve elements or aspects of interest to theuser (or to the system) so that the user (or the system) can completeoperations on such elements in cases where they would otherwise becomeinaccessible or unusable. As discussed in more detail below and in theclaims, other embodiments include a method or system capable of pausing,or otherwise modifying, the presentation of a scene or scene elements sothat elements of interest to a user that would otherwise becomeinaccessible are available for as long as they are needed for aninteraction.

Some method aspects of the present disclosure include (a) receiving arequest associated with an item, aspect, or element presented in ascene; (b) detecting that a first presentation of the item, aspect, orelement has left, or is about to leave, the field of view of the sceneor otherwise become inaccessible or difficult to access in the contextof a present activity; (c) preserving a presentation or proxypresentation associated with the item, aspect, or element by one or moreof, but not limited to (i) slowing the update or frame rate orpresentation rate of the scene or aspect of the scene; (ii)retaining/capturing or incorporating a presentation of the item, aspect,or element in the scene; (iii) generating a simulated presentation ofthe item, aspect, or element; or (iv) generating a proxy affordance forthe item, aspect, or element.

Additionally, embodiments may include (d) restoring the firstpresentation in response to one or more of (i) the end of theinaccessibility of the item in the context of the first presentation;(ii) user input; or (iii) the end of the present activity.

In one example, the present disclosure provides a way to slow down orpause a scene while the user interacts with items that would soon leaveor become obscured in a “live” scene, optionally followed by a processof catching up to the state of the live display.

Further aspects may include one or more of (in various combinations indifferent embodiments): (e) determining one or more scene presentationspecifications (e.g., rules for generating the scene; “live” vs.delayed, field of view, illumination, focus, highlighting, zoom, etc.);(f) determining presentations of interest, corresponding to one or moreitems, aspects, or elements of a scene based on one or more of (i) usertask; (ii) system task; (iii) context; (iv) user interest; (v) userpreferences; (g) determining a difficulty of interaction regarding thepresentations of interest, according to a first (current) scenepresentation specification (e.g., recognizing that an item will go offthe screen or move behind an obstacle or become small or hard to discernor touch if it follows its current trajectory or the update speed ismaintained or the user continues to move his device or position in thecurrent manner); (h) modifying aspects of of a first scene presentationspecification, and/or replacing a first scene presentation specificationwith a second scene presentation specification, which modifications orreplacements reduce the difficulty of interaction regarding thepresentations of interest; (i) restoring (e.g with animation or othertransitions) the first scene specification, and/or removingmodifications to the first scene specification in response to orprediction of (i) a determination of the end of a user's interest in orinteraction with a presentation of interest; (ii) a determination of areduction in difficulty of interaction regarding a presentation ofinterest employing the first scene presentation; (iii) a user request;(iv) at least one of a context, task or setting change; or (v)notification or interruption.

The present disclosure in some embodiments thus provides a way to modifythe presentation of items of interest in a scene that the rules forconstructing that scene would otherwise make inaccessible or unusable,e.g., by modifying the rules for generating the scene or aspects of thescene; followed, optionally, by restoring those rules.

One embodiment includes methods for pausing, capturing, or generatingpresentations associated with elements relevant to a current task orinteraction long enough for that task or interaction to be completed, insituations in which the present scene display would cause thosepresentations to become inaccessible or otherwise difficult to interactwith.

For example, a user might start to reserve a taxi by interacting withthe presentation of a taxi as it drives by, but be in the middle of thatinteraction as the cab speeds away or becomes obscured by a bus orbuilding, or drives into the distance and becomes too small to easilyinteract with on the screen. The present methods and systems may “pause”or “slow down” a scene or parts of a scene long enough for a user tocomplete her interaction, and then optionally “catch up” (e.g., bysomething like fast forward) to “live action” once the interaction iscomplete. In another embodiment, the present methods and systems mightenlarge the cab (if it has receded from view and become too small) orsimulate the removal of an obscuring object, such as another vehicle, abuilding, or a sign.

The present systems and methods also allow for “catching up” modified ordelayed scenes or presentations with live scene presentations, when thedelay or modifications are no longer necessary or desired. These aspectsare discussed above and may be used in scenarios in which the display ofa scene or aspects of a scene have been modified with respect to aninitial scene presentation specification. In particular, the presentmethods and systems may include a determination that certainpresentations or aspects of a presentation are being managed, modifiedor manipulated, and s “release” of those modifications or manipulationsin response to demands of task, context, or user input.

Additional aspects include tools for building applications and systemssupporting the features above, including platform elements, APIs, andclass frameworks that provide the relevant functionality, such as:“presentation needed” states; “presentation no longer needed” events;attributes characterizing the usability and accessibility of sceneelements (e.g., too small to touch, too fast-moving to track) for userswith physical or cognitive impairments, and associated events (e.g.,“object has become too small”), etc.

FIG. 15 illustrates an example augmented reality system 1522 in whichembodiments may be implemented. The system 1522 may work through anaugmented reality device 1502 for use by user 1500. Augmented realitysystem 1522 may be implemented on augmented reality device 1502, or itmay be implemented remotely, in whole or in part, for example as a cloudservice through network 1504 to communicate with augmented realitydevice 1502. Augmented reality device 1502 will have a visual field ofview 200 that will include real world object image data 1501 and realworld object motion data 1503.

The augmented reality system 1522 may contain, for example,environmental context evaluation module 1506, augmented reality devicecontext evaluation module 1508, request detection module 1510, objectdetection and tracking module 1511, object vector, speed, acceleration,and trajectory tracking module 1511, image presentation modificationmodule 1513, video manipulation module 1514, image database 1515,digital image generation module 1516, augmented reality representation1517, device field of view tracking module 1518, menu presentationmodule 1519, and/or presentation restoration module 1520. Augmentedreality system 322 running on or through augmented reality device 1502may communicate over a network 1504, wirelessly or by hardwiredconnection. By way of network 1504, which may include a cloud computingcomponent, augmented reality system 1522 may communicate to effectuatetransactions or other interactions with network payment system 1524,including credit account 1526, google wallet 1528, and/or PayPal 1530.Augmented reality system 1522 may also communicate to effectuatetransactions or other interactions via network 1504 with a retailer1532, such as Taxi company 1534 or an online retailer, such asAmazon.com 1535 or iTunes 1540. Augmented reality system 1522 may alsocommunicate to effectuate transactions or other interactions via network1504 with online data service 1536 such as Facebook 1538, iTunes 1540,and/or Google Play app store 1542.

In this way, a user may interact with digital representations of herenvironment in order to, inter alia, complete transactions, collectphysical or digital items of interest, e.g., order physical goods orcreate and transfer digital media including digital images of realobjects, or upload digital media to a social network such as Facebook orPinterest.

As referenced herein, the augmented reality system 1522 may be used toperform various data querying and/or recall techniques with respect toreal world objects and/or augmented reality representations of realworld objects. For example, where real world object image data isorganized, keyed to, and/or otherwise accessible using one or more imagedatabases, augmented reality system 1522 may employ various Boolean,statistical, and/or semi-boolean searching techniques to select acorrect real world object image among a set of images in a real worldscene, e.g., by request detection module 1510, and also to provide anaugmented reality representation 1517 of the object, either by findingone in, e.g., image database 1515, or by generating one, e.g., viadigital image generation module 1516.

Many examples of databases and database structures may be used inconnection with augmented reality system 1522. Such examples includehierarchical models (in which data is organized in a tree and/orparent-child node structure), network models (based on set theory, andin which multi-parent structures per child node are supported), orobject/relational models (combining the relational model with theobject-oriented model).

Still other examples include various types of eXtensible Mark-upLanguage (XML) databases. For example, a database may be included thatholds data in some format other than XML, but that is associated with anXML interface for accessing the database using XML. As another example,a database may store XML data directly. Additionally, or alternatively,virtually any semi-structured database may be used, so that context maybe provided to/associated with stored data elements (either encoded withthe data elements, or encoded externally to the data elements), so thatdata storage and/or access may be facilitated.

Such databases, and/or other memory storage techniques, may be writtenand/or implemented using various programming or coding languages. Forexample, object-oriented database management systems may be written inprogramming languages such as, for example, C++ or Java. Relationaland/or object/relational models may make use of database languages, suchas, for example, the structured query language (SQL), which may be used,for example, for interactive queries for disambiguating informationand/or for gathering and/or compiling data from the relationaldatabase(s).

For example, SQL or SQL-like operations over one or more real worldobject image data may be performed, or Boolean operations using realworld object image data 1501 may be performed. For example, weightedBoolean operations may be performed in which different weights orpriorities are assigned to one or more real world object imagesdepending on context of the scene or context of the device 1502,including other programs running on the device 1502, perhaps relative toone another. For example, a number-weighted, exclusive-OR operation maybe performed to request specific weightings of categories of objectsdepending on recognized cues such as known user preference, e.g.

In this way ambiguity in the complexity of selection of objects within ascene may be resolved, for example by ad hoc recognition of a categoryof items in view of the AR device that is known to be of interest to theuser. Such a recognition event by the system will greatly reduce theuniverse of objects in the scene from which to associate with anambiguous request, such as a gesture to a region of the AR device'sfield of view. In some embodiments, the system may make decisions as tothe exact nature of the user request in stages, for example byhighlighting successively smaller sets of objects and prompting the userto select from among them at each stage. This may involve nestedboundaries, for example, if a “Beatles” boundary is presented (asdiscussed in an example below, other, non-Beatles objects in the scenemay be removed (or the Beatles may be highlighted), then after selectionof Ringo Starr the other three Beatles could be removed, leaving Ringoat the requested item to be interacted with, signaling, perhaps,presentation of various menu options such as purchase music or movies,upload image or video data to a social network, or retrieve webinformation about Ringo Star.

In this way the system can distinguish location boundaries, such aspixel coordinates on an AR display, from semantic boundaries such as“Beatles.”

FIGS. 16-18 illustrate an example of a user interaction with anaugmented reality system that does not include the ability todynamically preserve elements in a scene as disclosed herein. FIG. 16depicts an augmented reality device (a smartphone) showing on itsdisplay the Beatles walking across Abbey Road. If the user wants to buysome music by Ringo Starr, her favorite Beatle, knowing that the ARapplication she is using supports music purchases for any item the ARapplication recognizes, she will have to be fast in tapping or otherwiserequesting interaction with Ringo's image to buy music.

FIG. 17 depicts a scenario in which the user has missed Ringo as hemoves across the display; he is too difficult for the user to select.(Or if the user does manage to select her target, it may leave thescreen before a desired action is taken, and context will be lost.).FIG. 18 depicts the same scene a few moments later when all of theBeatles have gone out of the field of view of the AR device, and off thescreen of the device.

FIGS. 19-23 depict the same scenario of viewing the Beatles crossingAbbey Road but this time with the presently disclosed technology fordynamically preserving elements of a scene implemented on or through theAR device.

FIG. 19 depicts again a user trying to tap on Ringo as his image travelsacross the screen of an AR device.

FIG. 20 depicts a successful “tap” on Ringo as the system recognizes hisimage as an item of interest to the user, perhaps by both the tap as anindicator and by virtue of a previously expressed interest in RingoStarr, known to the system, for example stored in environmental contextevaluation module 1506, which can evaluate recognizable objects from theenvironment and match them to objects in a stored database of images.The successful tap and recognition by the system that it represents auser “request” for interaction with, in this case the person selected,may coincide with a vector physics analysis of Ringo Starr's real worldmovement relative to the field of view of the AR device. Such ananalysis may be carried out by, e.g., object detection and trackingmodule 1511 and/or object vector, speed, acceleration, and trajectoryprocessing module 1512. This analysis may be carried out in two or threedimensions, and it may factor in time as well, for example time untilthe object of interest is no longer within the field of view of the ARdevice, and therefore not available for interaction on the AR device.

Here, the augmented reality system 1522 may contain one or morethresholds for calculated time periods, for example, beneath which itwill pause the requested item on the display. For example, a thresholdfor interacting with an element of a scene of 5 seconds may beprogrammed into the image presentation modification module 1513; ifsubsequent to the request, object vector, speed, acceleration, andtrajectory processing module 1512 calculates that at Ringo's currentrate of speed and current bearing, Ringo's image will leave the ARdisplay in 1.5 seconds, then that may trigger video manipulation module1514 to freeze or slow the video of Ringo's crossing on the display toallow the user to perform the desired interaction (because it we belowthe 5 second threshold lower limit). Similar thresholds may operate asto the size of the object on the display, for example an object thatbecomes very small, e.g., less than one square centimeter, may be deemedno longer viable for interaction and accordingly enlarged forinteraction, e.g., by digital image generation module 1516 to create alarger, augmented reality representation 1517 of the object, perhapswith associated menus or command buttons to denote and facilitate theavailable interactions.

As shown in FIG. 20, the AR system may highlight the selected object asa way of confirming with the user that the correct request hasregistered with the system, and that the item of interest has been“paused” or otherwise augmented (e.g. frozen, slowed down, de-obscured,enlarged, or otherwise made more suitable for an interaction on the ARdevice).

As shown in FIG. 21, even though time continues to flow around Ringo(his bandmates have moved offscreen, cars continue to drive on the road,et cetera), Ringo has been frozen in place. This guarantees that theuser is able to act on this particular item of interest in the scene; itremains viable for interaction for a longer period of time than itotherwise would have if the scene was presented “live.”

As shown in FIG. 22, the system now has time to identify Ringo to theuser and to tag him with several commands or menu options that areavailable, including, for example, “Buy Music,” “Find Pictures,” or“post image or video to FaceBook.”

As shown in FIG. 23, optionally in some embodiments, when the user is nolonger interested in Ringo, she releases him, and we see Ringo“fast-forward”—in this case, he rushes off-screen to catch up with hisbandmates—after which the entire AR-scene is shown “live” again.

FIGS. 24-30 illustrate operational flows representing example operationsrelated to dynamically preserving elements in augmented reality systems.In these figures that include various examples of operational flows,discussion and explanation may be provided with respect to theabove-described system environment of FIG. 15, and/or with respect toother examples and contexts. However, it should be understood that theoperational flows may be executed in a number of other environments andcontexts and/or in modified versions of FIGS. 15-23. Also, although thevarious operational flows are presented in the sequence(s) illustrated,it should be understood that the various operations may be performed inother orders than those which are illustrated, or may be performedconcurrently.

In one embodiment, augmented reality system 1522 may include circuitryfor accepting a user request associated with at least one of an item, anaspect, or an element of a field of view of an augmented reality device;circuitry for determining that a first presentation of the at least oneitem, aspect, or element has a limited period of viability for userinteraction relative to the field of view of the augmented realitydevice; and circuitry for at least one of maintaining the firstpresentation or providing a substantially similar second presentation inresponse to at least one output of the circuitry for determining that afirst presentation of the at least one item, aspect, or element has alimited period of viability for interaction relative to the field ofview of the augmented reality device.

Temporal Element Restoration

The present disclosure provides a system in which elements of a scenepresented on an AR device may be modified and/or transformed in waysthat preserve elements or aspects of interest to the user (or to thesystem) so that the user (or the system) can complete operations on suchelements in cases where they would otherwise be inaccessible orunavailable.

The present disclosure includes two related but distinct subsystems thatmay be deployed together in typical embodiments: (1) methods and systemsfor making elements of interest that are no longer visible in the “live”scene available in, or accessible from, that scene, or a modified scene;and optionally (2) methods and systems for removing the modified ordelayed scenes or presentations, and restoring live scene presentationswhen the modifications are no longer necessary.

Some aspects of the present disclosure include: (a) receiving a requestassociated with an item, aspect, or element not presented in the currentscene, including (i) notifications pertaining to the item, aspect, orelement; (ii) content or actions related to the item, aspect, orelement; (ii) system state (e.g., of a launched or restored application,tool, or process) referring to, employing, and/or containing the item,aspect, or element; (b) producing (in response to the request) arepresentation associated with the item, aspect, or element, includingone or more of (i) a substitution of a related item, aspect, or element(currently) present in the scene; (ii) a generation of a(scene-appropriate) proxy representation of the item, aspect, orelement; (iii) a modification of the scene to include a (suitable) proxyrepresentation of the item, aspect, or element; (c) processing therequest and any subsequent related actions through subsequentinteractions with the produced representation.

An alternative embodiment provides for (d) a process for acquiring (inresponse to the request) a suitable representation, includingsuggestions or hints to the user for operating his device or otherwisetaking actions that will bring a suitable representation (including butnot limited to the original item, aspect, or element) into the presentscene.

Further embodiments include (e) terminating, dismissing, removing, orreleasing the representation in response to completion of the request oraspects of the request, or in response to direction by the user.Finally, typical embodiments will provide (f) distinctive (optionallydisplayed) indications and affordances associated with produced (b) oracquired (d) items.

Another embodiment may include receiving a request associated with anitem or aspect previously presented in an AR scene that is no longerpresented in the present scene (e.g., special highlighting, etc.) and byfurther constraining the previous and present scenes to a singleapplication, system, session, or timeframe.

In an example embodiment, the user might preform some task—such asswitching to a new application, or responding to an e-mail ornotification, or opening a document—that requires or references an itemthat is no longer visible in or accessible from the current “live” ARscene on his device. In these cases, the system may provide indications,in the present scene, of actions the user can take to “acquire” the item(e.g., where he should point his device), or the system may present theitem as if it were a part of the current scene, temporarily inserting oroverlaying it for the duration of the relevant operations, or the systemmay modify the scene so that the item appears to actually be a part ofthe scene, or the system may substitute a related item present in thescene as a proxy for the “missing” item.

For example, the user might use an AR street-scene application toreserve a taxi by interacting with a cab as it drives by, receiving arequest for confirmation and additional details only after the taxi hasdriven out of range of scenes he can capture through his augmentedreality device. In response to the subsequent receipt of a confirmationor other information regarding the cab, the presently disclosed systemmight then use another taxi, or a simulated taxi to represent themissing cab and to serve as a target for the subsequent interactionrelated to the transaction initiated with the missing cab.

In a more extreme and somewhat whimsical example that illustrates thepoint, the user might have entered a toy store when the request tocontinue the transaction arises, in which case the proxy for the missingcab might be a toy taxi in the store, or a synthesized item rendered asa toy taxi on the shelf of the store.

Alternatively, if a taxi (perhaps even the originally-designated cab) isnearby or can easily be acquired by the user's device, the system mightshow an indication of the direction in which the user needs to point hisdevice to reacquire the target cab.

In some embodiments, an object no longer present that relates to arequest to the user for interaction may be presented on the augmentedreality device in a different way than the original object appeared,though perhaps related. For example, in the taxi example, if a requestto confirm a taxi reservation is received by a user inside a building,an augmented reality representation of the taxi cab may not be the cabitself, but rather a computer generated image of a cab driver with thecab company logo on his clothing. In this way the request may bepresented to the user in a way that is appropriate to the context of theuser and the device. The device may automatically present an appropriaterepresentation based on detected context, for example via environmentalcontext evaluation module 3106. Rulesets such as “no cars insidebuildings, instead present a person” may be used to facilitate thisfunction.

Optionally, one embodiment inludes systems and methods for removingmodified or delayed scenes or presentations, and reverting to a livepresentation of the field of view of the augmented reality device whenthe modifications are no longer necessary. These embodiments includesituations in which the display of a scene or aspects of a scene havebeen modified with respect to an initial scene presentationspecification. In particular, one embodiment includes a determinationthat certain presentations or aspects of a presentation are beingmodified, and a “release” of those modifications in response to demandsof task, context, or user input.

In another example of context-specific presentation of a request, a userreading a newspaper through augmented reality glasses may be presentedwith a request for confirmation of the taxi cab reservation via avirtual post-it note containing the cab company's logo situated on thenewspaper, as an augmented reality representation of the taxi cab thatwas earlier selected to make the reservation.

Additional aspects of the present disclosure include tools for buildingapplications and systems supporting the features above, includingplatform elements, APIs, and class frameworks that provide the relevantfunctionality.

FIG. 31 illustrates an example augmented reality system 3122 in whichembodiments may be implemented. The system 3122 may work through anaugmented reality device 3102 for use by user 3100. Augmented realitysystem 3122 may be implemented on augmented reality device 3102, or itmay be implemented remotely via a network 3104, in whole or in part, forexample as a cloud service through network 3104 to communicate withaugmented reality device 3102. Augmented reality device 3102 will have avisual field of view 200 that will include real world object image data3101 and real world object motion data 3103.

The augmented reality system 3122 may contain, for example,environmental context evaluation module 3106, augmented reality devicecontext evaluation module 3108, request detection module 3110, objectdetection and tracking module 3111, object vector, speed, acceleration,and trajectory tracking module 3112, image presentation modificationmodule 3113, video manipulation module 3114, image database 3115,digital image generation module 3116, augmented reality representation3117, device field of view tracking module 3118, menu presentationmodule 3119, presentation restoration module 3120, and/or requestprocessing module 3121.

Augmented reality system 3122 running on or through augmented realitydevice 3102 may communicate over a network 3104 wirelessly or byhardwired connection. By way of network 3104, which may include a cloudcomputing component, augmented reality system 3122 may communicate toeffectuate transactions or other interactions with network paymentsystem 3124, including credit account 3126, google wallet 3128, and/orPayPal 3130. Augmented reality system 3122 may also communicate toeffectuate transactions or other interactions via network 3104 with aretailer 3132, such as Taxi company 3134 or an online retailer, such asAmazon.com 3135 or iTunes 3140. Augmented reality system 3122 may alsocommunicate to effectuate transactions or other interactions via network3104 with online data service 3136 such as Facebook 3138, iTunes 3140,and/or Google Play app store 3142.

In this way, a user may interact with digital representations of herenvironment in order to, inter alia, track orders of physical goods orservices, complete transactions, or conduct lengthy or discontinuouscommunications.

As referenced herein, the augmented reality system 3122 may be used toperform various data querying and/or recall techniques with respect toreal world objects and/or augmented reality representations of realworld objects. For example, where real world object image data isorganized, keyed to, and/or otherwise accessible using one or more imagedatabases, augmented reality system 3122 may employ various Boolean,statistical, and/or semi-boolean searching techniques to select acorrect real world object image among a set of images in a real worldscene, e.g., by request detection module 3110, and also to provide anaugmented reality representation 3117 of the object, either by findingone in, e.g., image database 3115, or by generating one, e.g., viadigital image generation module 3116.

Many examples of databases and database structures may be used inconnection with augmented reality system 3122. Such examples includehierarchical models (in which data is organized in a tree and/orparent-child node structure), network models (based on set theory, andin which multi-parent structures per child node are supported), orobject/relational models (combining the relational model with theobject-oriented model).

Still other examples include various types of eXtensible Mark-upLanguage (XML) databases. For example, a database may be included thatholds data in some format other than XML, but that is associated with anXML interface for accessing the database using XML. As another example,a database may store XML data directly. Additionally, or alternatively,virtually any semi-structured database may be used, so that context maybe provided to/associated with stored data elements (either encoded withthe data elements, or encoded externally to the data elements), so thatdata storage and/or access may be facilitated.

Such databases, and/or other memory storage techniques, may be writtenand/or implemented using various programming or coding languages. Forexample, object-oriented database management systems may be written inprogramming languages such as, for example, C++ or Java. Relationaland/or object/relational models may make use of database languages, suchas, for example, the structured query language (SQL), which may be used,for example, for interactive queries for disambiguating informationand/or for gathering and/or compiling data from the relationaldatabase(s).

For example, SQL or SQL-like operations over one or more real worldobject image data may be performed, or Boolean operations using realworld object image data 3101 may be performed. For example, weightedBoolean operations may be performed in which different weights orpriorities are assigned to one or more real world object imagesdepending on context of the scene or context of the device 3102,including other programs running on the device 3102, perhaps relative toone another. For example, a number-weighted, exclusive-OR operation maybe performed to request specific weightings of categories of objectsdepending on recognized cues such as known user preference, specificrulesets in place to define relationships between objects or objecttypes and supported communicatinons and/or transactions that can beinitiated, mediated, and/or completed by the augmented reality system3122 and/or augmented reality device 3102.

In this way, a call-and-response kind of interaction can be carried outwherein a call by a user (e.g., a request for a taxi cab reservationmanifested as a tap (or, e.g., a detected dwell by an eyetrackingdevice) on a cab in a field of view of an AR device) can be processedand a response brought back to the user at a later time on the AR devicethrough a “request” to the system to engage the user via display of anaugmented reality representation of the original taxi cab or an aspectof it, wherein the actual taxi cab is no longer in the field of view ofthe AR device.

In this way the system can track interactions that involve lengthy ordiscontinuous communications such as delivery tracking; orderfulfillment; simple email, text, or voice messaging; or scheduling.

FIGS. 32-39 illustrate an example of a user interaction with anaugmented reality device and system that implements temporal elementrestoration as disclosed herein. FIG. 32 depicts an augmented realitydevice (a smartphone) showing on its display a scene including a taxicab. FIG. 33 depicts a user's finger tapping the image of the taxi cabon the display of the augmented reality device 3102. FIG. 34 depicts twocommand options that the system has placed on the AR display in responseto the selection of the taxi cab: “Speedy Taxi Co.” and “Reserve Taxi”are displayed near the cab as graphical buttons on the display. FIG. 35depicts the user's finger tapping the “Reserve Taxi” button to reservethe taxi.

FIG. 36 depicts a user observing a street through the AR device at atime when the taxi is no longer present in the scene. The taxi companynow wants to confirm the reservation the user made earlier, but thereare no taxis on the street to provide the user with context for theconfirmation. FIG. 37 depicts a way in which the augmented realitysystem 3122 may provide the user with appropriate context (and to offerthe user the appropriate affordances for interaction to complete therequest for confirmation from the Taxi company), wherein the AR-enableddevice and/or the AR system generates an augmented realityrepresentation 3117 of the taxi or a “virtual taxi” to place within theAR scene. This provides a context for the user to grasp in addressingthe request for confirmation of the reservation from the Taxi company.

A request may be detected by request detection module 3110, and digitalimage generation module may automatically present based on the detectedrequest, e.g., either an image of the object relevant to the request, amodified image of the object, a computer generated version of theobject, or a different object that is related to the object.

FIG. 38 depicts the user's finger tapping on the virtual taxi to selectit, and FIG. 39 depicts command options that have been placed by thesystem near the virtual taxi as items that the user can activate, e.g.,by tapping to confirm the reservation.

Optionally, as depicted in FIG. 40, once confirmation is made, thesystem may remove the “virtual taxi” from the AR scene, e.g., therebyrestoring a “live” view of the scene.

FIGS. 41-47 illustrate operational flows representing example operationsrelated to temporal element restoration in augmented reality systems. Inthese figures that include various examples of operational flows,discussion and explanation may be provided with respect to theabove-described system environment of FIG. 31, and/or with respect toother examples and contexts. However, it should be understood that theoperational flows may be executed in a number of other environments andcontexts and/or in modified versions of FIGS. 31-40. Also, although thevarious operational flows are presented in the sequence(s) illustrated,it should be understood that the various operations may be performed inother orders than those which are illustrated, or may be performedconcurrently.

Operational/Functional Language Herein Describes Machines/MachineControl/Machine-Controlled Processes Unless Context Dictates Otherwise

The claims, description, and drawings of this application may describeone or more of the instant technologies in operational/functionallanguage, for example as a set of operations to be performed by acomputer. Such operational/functional description in most instanceswould be understood by one skilled in the art as specifically configuredhardware (e.g., because a general purpose computer in effect becomes aspecial purpose computer once it is programmed to perform particularfunctions pursuant to instructions from program software).

Importantly, although the operational/functional descriptions describedherein are understandable by the human mind, they are not abstract ideasof the operations/functions divorced from computational implementationof those operations/functions. Rather, the operations/functionsrepresent a specification for massively complex computational machinesor other means. As discussed in detail below, the operational/functionallanguage must be read in its proper technological context, i.e., asconcrete specifications for physical implementations. The logicaloperations/functions described herein are a distillation of machinespecifications or other physical mechanisms specified by theoperations/functions such that the otherwise inscrutable machinespecifications may be comprehensible to the human reader. Thedistillation also allows one of skill in the art to adapt theoperational/functional description of the technology across manydifferent specific vendors' hardware configurations or platforms,without being limited to specific vendors' hardware configurations orplatforms.

Some of the present technical description (e.g., detailed description,drawings, claims, etc.) may be set forth in terms of logicaloperations/functions. As described in more detail herein, these logicaloperations/functions are not representations of abstract ideas, butrather are representative of static or sequenced specifications ofvarious hardware elements. Differently stated, unless context dictatesotherwise, the logical operations/functions will be understood by thoseof skill in the art to be representative of static or sequencedspecifications of various hardware elements. This is true because toolsavailable to one of skill in the art to implement technical disclosuresset forth in operational/functional formats—tools in the form of a highlevel programming language (e.g., C, java, visual basic, etc.), or toolsin the form of Very High speed Hardware Description Language (“VHDL,”which is a language that uses text to describe logic circuits)—aregenerators of static or sequenced specifications of various hardwareconfigurations. This fact is sometimes obscured by the broad term“software,” but, as shown by the following explanation, those skilled inthe art understand that what is termed “software” is a shorthand for amassively complex interchaining/specification of ordered matterelements. The term “ordered matter elements” may refer to physicalcomponents of computation, such as assemblies of electronic logic gates,molecular computing logic constituents, quantum computing mechanisms,etc. For example, a high level programming language is a programminglanguage with strong abstraction, e.g., multiple levels of abstraction,from the details of the sequential organizations, states, inputs,outputs, etc., of the machines that a highlevel programming languageactually specifies. See, e.g., Wikipedia, High-level programminglanguage, http://en.wikipedia.org/wiki/Highlevel_programming_language(as of Jun. 5, 2012, 21:00 GMT). In order to facilitate humancomprehension, in many instances, high level programming languagesresemble or even share symbols with natural languages. See, e.g.,Wikipedia, Natural language,http://en.wikipedia.org/wiki/Natural_language (as of Jun. 5, 2012, 21:00GMT).

It has been argued that because high level programming languages usestrong abstraction (e.g., that they may resemble or share symbols withnatural languages), they are therefore a “purely mental construct”(e.g., that “software”—a computer program or computer programming—issomehow an ineffable mental construct, because at a high level ofabstraction, it can be conceived and understood by a human reader). Thisargument has been used to characterize technical description in the formof functions/operations as somehow “abstract ideas.” In fact, intechnological arts (e.g., the information and communicationtechnologies) this is not true.

The fact that high-level programming languages use strong abstraction tofacilitate human understanding should not be taken as an indication thatwhat is expressed is an abstract idea. In fact, those skilled in the artunderstand that just the opposite is true. If a high-level programminglanguage is the tool used to implement a technical disclosure in theform of functions/operations, those skilled in the art will recognizethat, far from being abstract, imprecise, “fuzzy,” or “mental” in anysignificant semantic sense, such a tool is instead a nearincomprehensibly precise sequential specification of specificcomputational machines—the parts of which are built up byactivating/selecting such parts from typically more generalcomputational machines over time (e.g., clocked time). This fact issometimes obscured by the superficial similarities between high-levelprogramming languages and natural languages. These superficialsimilarities also may cause a glossing over of the fact that high levelprogramming language implementations ultimately perform valuable work bycreating/controlling many different computational machines.

The many different computational machines that a high-level programminglanguage specifies are almost unimaginably complex. At base, thehardware used in the computational machines typically consists of sometype of ordered matter (e.g., traditional electronic devices (e.g.,transistors), deoxyribonucleic acid (DNA), quantum devices, mechanicalswitches, optics, fluidics, pneumatics, optical devices (e.g., opticalinterference devices), molecules, etc.) that are arranged to form logicgates. Logic gates are typically physical devices that may beelectrically, mechanically, chemically, or otherwise driven to changephysical state in order to create a physical reality of logic, such asBoolean logic.

Logic gates may be arranged to form logic circuits, which are typicallyphysical devices that may be electrically, mechanically, chemically, orotherwise driven to create a physical reality of certain logicalfunctions. Types of logic circuits include such devices as multiplexers,registers, arithmetic logic units (ALUs), computer memory, etc., eachtype of which may be combined to form yet other types of physicaldevices, such as a central processing unit (CPU)—the best known of whichis the microprocessor. A modern microprocessor will often contain morethan one hundred million logic gates in its many logic circuits (andoften more than a billion transistors). See, e.g., Wikipedia, Logicgates, http://en.wikipedia.org/wiki/Logic_gates (as of Jun. 5, 2012,21:03 GMT).

The logic circuits forming the microprocessor are arranged to provide amicroarchitecture that will carry out the instructions defined by thatmicroprocessor's defined Instruction Set Architecture. The InstructionSet Architecture is the part of the microprocessor architecture relatedto programming, including the native data types, instructions,registers, addressing modes, memory architecture, interrupt andexception handling, and external Input/Output. See, e.g., Wikipedia,Computer architecture,http://en.wikipedia.org/wiki/Computer_architecture (as of Jun. 5, 2012,21:03 GMT).

The Instruction Set Architecture includes a specification of the machinelanguage that can be used by programmers to use/control themicroprocessor. Since the machine language instructions are such thatthey may be executed directly by the microprocessor, typically theyconsist of strings of binary digits, or bits. For example, a typicalmachine language instruction might be many bits long (e.g., 32, 64, or128 bit strings are currently common). A typical machine languageinstruction might take the form “11110000101011110000111100111111” (a 32bit instruction). It is significant here that, although the machinelanguage instructions are written as sequences of binary digits, inactuality those binary digits specify physical reality. For example, ifcertain semiconductors are used to make the operations of Boolean logica physical reality, the apparently mathematical bits “1” and “0” in amachine language instruction actually constitute a shorthand thatspecifies the application of specific voltages to specific wires. Forexample, in some semiconductor technologies, the binary number “1”(e.g., logical “1”) in a machine language instruction specifies around+5 volts applied to a specific “wire” (e.g., metallic traces on aprinted circuit board) and the binary number “0” (e.g., logical “0”) ina machine language instruction specifies around −5 volts applied to aspecific “wire.” In addition to specifying voltages of the machines'configurations, such machine language instructions also select out andactivate specific groupings of logic gates from the millions of logicgates of the more general machine. Thus, far from abstract mathematicalexpressions, machine language instruction programs, even though writtenas a string of zeros and ones, specify many, many constructed physicalmachines or physical machine states.

Machine language is typically incomprehensible by most humans (e.g., theabove example was just ONE instruction, and some personal computersexecute more than two billion instructions every second). See, e.g.,Wikipedia, Instructions per second,http://en.wikipedia.org/wiki/Instructions_per_second (as of Jun. 5,2012, 21:04 GMT). Thus, programs written in machine language—which maybe tens of millions of machine language instructions long—areincomprehensible to most humans. In view of this, early assemblylanguages were developed that used mnemonic codes to refer to machinelanguage instructions, rather than using the machine languageinstructions' numeric values directly (e.g., for performing amultiplication operation, programmers coded the abbreviation “mult,”which represents the binary number “011000” in MIPS machine code). Whileassembly languages were initially a great aid to humans controlling themicroprocessors to perform work, in time the complexity of the work thatneeded to be done by the humans outstripped the ability of humans tocontrol the microprocessors using merely assembly languages.

At this point, it was noted that the same tasks needed to be done overand over, and the machine language necessary to do those repetitivetasks was the same. In view of this, compilers were created. A compileris a device that takes a statement that is more comprehensible to ahuman than either machine or assembly language, such as “add 2+2 andoutput the result,” and translates that human understandable statementinto a complicated, tedious, and immense machine language code (e.g.,millions of 32, 64, or 128 bit length strings). Compilers thus translatehigh-level programming language into machine language. This compiledmachine language, as described above, is then used as the technicalspecification which sequentially constructs and causes theinteroperation of many different computational machines such thatuseful, tangible, and concrete work is done. For example, as indicatedabove, such machine language—the compiled version of the higher-levellanguage—functions as a technical specification which selects outhardware logic gates, specifies voltage levels, voltage transitiontimings, etc., such that the useful work is accomplished by thehardware.

Thus, a functional/operational technical description, when viewed by oneof skill in the art, is far from an abstract idea. Rather, such afunctional/operational technical description, when understood throughthe tools available in the art such as those just described, is insteadunderstood to be a humanly understandable representation of a hardwarespecification, the complexity and specificity of which far exceeds thecomprehension of most any one human. With this in mind, those skilled inthe art will understand that any such operational/functional technicaldescriptions—in view of the disclosures herein and the knowledge ofthose skilled in the art—may be understood as operations made intophysical reality by (a) one or more interchained physical machines, (b)interchained logic gates configured to create one or more physicalmachine(s) representative of sequential/combinatorial logic(s), (c)interchained ordered matter making up logic gates (e.g., interchainedelectronic devices (e.g., transistors), DNA, quantum devices, mechanicalswitches, optics, fluidics, pneumatics, molecules, etc.) that createphysical reality of logic(s), or (d) virtually any combination of theforegoing. Indeed, any physical object which has a stable, measurable,and changeable state may be used to construct a machine based on theabove technical description. Charles Babbage, for example, constructedthe first mechanized computational apparatus out of wood with themechanism powered by cranking a handle.

Thus, far from being understood as an abstract idea, those skilled inthe art will recognize a functional/operational technical description asa humanlyunderstandable representation of one or more almostunimaginably complex and time sequenced hardware instantiations. Thefact that functional/operational technical descriptions might lendthemselves readily to high-level computing languages (or high-levelblock diagrams for that matter) that share some words, structures,phrases, etc. with natural language should not be taken as an indicationthat such functional/operational technical descriptions are abstractideas, or mere expressions of abstract ideas. In fact, as outlinedherein, in the technological arts this is simply not true. When viewedthrough the tools available to those of skill in the art, suchfunctional/operational technical descriptions are seen as specifyinghardware configurations of almost unimaginable complexity.

As outlined above, the reason for the use of functional/operationaltechnical descriptions is at least twofold. First, the use offunctional/operational technical descriptions allows near-infinitelycomplex machines and machine operations arising from interchainedhardware elements to be described in a manner that the human mind canprocess (e.g., by mimicking natural language and logical narrativeflow). Second, the use of functional/operational technical descriptionsassists the person of skill in the art in understanding the describedsubject matter by providing a description that is more or lessindependent of any specific vendor's piece(s) of hardware.

The use of functional/operational technical descriptions assists theperson of skill in the art in understanding the described subject mattersince, as is evident from the above discussion, one could easily,although not quickly, transcribe the technical descriptions set forth inthis document as trillions of ones and zeroes, billions of single linesof assembly-level machine code, millions of logic gates, thousands ofgate arrays, or any number of intermediate levels of abstractions.However, if any such low-level technical descriptions were to replacethe present technical description, a person of skill in the art couldencounter undue difficulty in implementing the disclosure, because sucha low-level technical description would likely add complexity without acorresponding benefit (e.g., by describing the subject matter utilizingthe conventions of one or more vendor-specific pieces of hardware).

Thus, the use of functional/operational technical descriptions assiststhose of skill in the art by separating the technical descriptions fromthe conventions of any vendor-specific piece of hardware.

In view of the foregoing, the logical operations/functions set forth inthe present technical description are representative of static orsequenced specifications of various ordered-matter elements, in orderthat such specifications may be comprehensible to the human mind andadaptable to create many various hardware configurations. The logicaloperations/functions disclosed herein should be treated as such, andshould not be disparagingly characterized as abstract ideas merelybecause the specifications they represent are presented in a manner thatone of skill in the art can readily understand and apply in a mannerindependent of a specific vendor's hardware implementation.

One skilled in the art will recognize that the herein describedcomponents (e.g., operations), devices, objects, and the discussionaccompanying them are used as examples for the sake of conceptualclarity and that various configuration modifications are contemplated.Consequently, as used herein, the specific exemplars set forth and theaccompanying discussion are intended to be representative of their moregeneral classes. In general, use of any specific exemplar is intended tobe representative of its class, and the non-inclusion of specificcomponents (e.g., operations), devices, and objects should not be takenas limiting.

Although a user may be shown/described herein as a single illustratedfigure, those skilled in the art will appreciate that any user may berepresentative of a human user, a robotic user (e.g., computationalentity), and/or substantially any combination thereof (e.g., a user maybe assisted by one or more robotic agents) unless context dictatesotherwise. Those skilled in the art will appreciate that, in general,the same may be said of “sender” and/or other entity-oriented terms assuch terms are used herein unless context dictates otherwise.

Those skilled in the art will appreciate that the foregoing specificexemplary processes and/or devices and/or technologies arerepresentative of more general processes and/or devices and/ortechnologies taught elsewhere herein, such as in the claims filedherewith and/or elsewhere in the present application.

Those having skill in the art will recognize that the state of the arthas progressed to the point where there is little distinction leftbetween hardware and software implementations of aspects of systems; theuse of hardware or software is generally (but not always, in that incertain contexts the choice between hardware and software can becomesignificant) a design choice representing cost vs. efficiency tradeoffs.Those having skill in the art will appreciate that there are variousvehicles by which processes and/or systems and/or other technologiesdescribed herein can be effected (e.g., hardware, software, and/orfirmware), and that the preferred vehicle will vary with the context inwhich the processes and/or systems and/or other technologies aredeployed. For example, if an implementer determines that speed andaccuracy are paramount, the implementer may opt for a mainly hardwareand/or firmware vehicle; alternatively, if flexibility is paramount, theimplementer may opt for a mainly software implementation; or, yet againalternatively, the implementer may opt for some combination of hardware,software, and/or firmware. Hence, there are several possible vehicles bywhich the processes and/or devices and/or other technologies describedherein may be effected, none of which is inherently superior to theother in that any vehicle to be utilized is a choice dependent upon thecontext in which the vehicle will be deployed and the specific concerns(e.g., speed, flexibility, or predictability) of the implementer, any ofwhich may vary. Those skilled in the art will recognize that opticalaspects of implementations will typically employ optically-orientedhardware, software, and or firmware.

In some implementations described herein, logic and similarimplementations may include software or other control structures.Electronic circuitry, for example, may have one or more paths ofelectrical current constructed and arranged to implement variousfunctions as described herein. In some implementations, one or moremedia may be configured to bear a device-detectable implementation whensuch media hold or transmit a device detectable instructions operable toperform as described herein. In some variants, for example,implementations may include an update or modification of existingsoftware or firmware, or of gate arrays or programmable hardware, suchas by performing a reception of or a transmission of one or moreinstructions in relation to one or more operations described herein.Alternatively or additionally, in some variants, an implementation mayinclude special-purpose hardware, software, firmware components, and/orgeneral-purpose components executing or otherwise invokingspecial-purpose components. Specifications or other implementations maybe transmitted by one or more instances of tangible transmission mediaas described herein, optionally by packet transmission or otherwise bypassing through distributed media at various times.

Alternatively or additionally, implementations may include executing aspecial-purpose instruction sequence or invoking circuitry for enabling,triggering, coordinating, requesting, or otherwise causing one or moreoccurrences of virtually any functional operations described herein. Insome variants, operational or other logical descriptions herein may beexpressed as source code and compiled or otherwise invoked as anexecutable instruction sequence. In some contexts, for example,implementations may be provided, in whole or in part, by source code,such as C++, or other code sequences. In other implementations, sourceor other code implementation, using commercially available and/ortechniques in the art, may be compiled//implemented/translated/convertedinto a high-level descriptor language (e.g., initially implementingdescribed technologies in C or C++ programming language and thereafterconverting the programming language implementation into alogic-synthesizable language implementation, a hardware descriptionlanguage implementation, a hardware design simulation implementation,and/or other such similar mode(s) of expression). For example, some orall of a logical expression (e.g., computer programming languageimplementation) may be manifested as a Verilog-type hardware description(e.g., via Hardware Description Language (HDL) and/or Very High SpeedIntegrated Circuit Hardware Descriptor Language (VHDL)) or othercircuitry model which may then be used to create a physicalimplementation having hardware (e.g., an Application Specific IntegratedCircuit). Those skilled in the art will recognize how to obtain,configure, and optimize suitable transmission or computational elements,material supplies, actuators, or other structures in light of theseteachings.

The foregoing detailed description has set forth various embodiments ofthe devices and/or processes via the use of block diagrams, flowcharts,and/or examples. Insofar as such block diagrams, flowcharts, and/orexamples contain one or more functions and/or operations, it will beunderstood by those within the art that each function and/or operationwithin such block diagrams, flowcharts, or examples can be implemented,individually and/or collectively, by a wide range of hardware, software,firmware, or virtually any combination thereof. In one embodiment,several portions of the subject matter described herein may beimplemented via Application Specific Integrated Circuits (ASICs), FieldProgrammable Gate Arrays (FPGAs), digital signal processors (DSPs), orother integrated formats. However, those skilled in the art willrecognize that some aspects of the embodiments disclosed herein, inwhole or in part, can be equivalently implemented in integratedcircuits, as one or more computer programs running on one or morecomputers (e.g., as one or more programs running on one or more computersystems), as one or more programs running on one or more processors(e.g., as one or more programs running on one or more microprocessors),as firmware, or as virtually any combination thereof, and that designingthe circuitry and/or writing the code for the software and or firmwarewould be well within the skill of one of skill in the art in light ofthis disclosure. In addition, those skilled in the art will appreciatethat the mechanisms of the subject matter described herein are capableof being distributed as a program product in a variety of forms, andthat an illustrative embodiment of the subject matter described hereinapplies regardless of the particular type of signal bearing medium usedto actually carry out the distribution. Examples of a signal bearingmedium include, but are not limited to, the following: a recordable typemedium such as a floppy disk, a hard disk drive, a Compact Disc (CD), aDigital Video Disk (DVD), a digital tape, a computer memory, etc.; and atransmission type medium such as a digital and/or an analogcommunication medium (e.g., a fiber optic cable, a waveguide, a wiredcommunications link, a wireless communication link (e.g., transmitter,receiver, transmission logic, reception logic, etc.), etc.).

In a general sense, those skilled in the art will recognize that thevarious aspects described herein which can be implemented, individuallyand/or collectively, by a wide range of hardware, software, firmware,and/or any combination thereof can be viewed as being composed ofvarious types of “electrical circuitry.” Consequently, as used herein“electrical circuitry” includes, but is not limited to, electricalcircuitry having at least one discrete electrical circuit, electricalcircuitry having at least one integrated circuit, electrical circuitryhaving at least one application specific integrated circuit, electricalcircuitry forming a general purpose computing device configured by acomputer program (e.g., a general purpose computer configured by acomputer program which at least partially carries out processes and/ordevices described herein, or a microprocessor configured by a computerprogram which at least partially carries out processes and/or devicesdescribed herein), electrical circuitry forming a memory device (e.g.,forms of memory (e.g., random access, flash, read only, etc.)), and/orelectrical circuitry forming a communications device (e.g., a modem,communications switch, optical-electrical equipment, etc.). Those havingskill in the art will recognize that the subject matter described hereinmay be implemented in an analog or digital fashion or some combinationthereof.

Those skilled in the art will recognize that at least a portion of thedevices and/or processes described herein can be integrated into a dataprocessing system. Those having skill in the art will recognize that adata processing system generally includes one or more of a system unithousing, a video display device, memory such as volatile or non-volatilememory, processors such as microprocessors or digital signal processors,computational entities such as operating systems, drivers, graphicaluser interfaces, and applications programs, one or more interactiondevices (e.g., a touch pad, a touch screen, an antenna, etc.), and/orcontrol systems including feedback loops and control motors (e.g.,feedback for sensing position and/or velocity; control motors for movingand/or adjusting components and/or quantities). A data processing systemmay be implemented utilizing suitable commercially available components,such as those typically found in data computing/communication and/ornetwork computing/communication systems.

Those skilled in the art will recognize that it is common within the artto implement devices and/or processes and/or systems, and thereafter useengineering and/or other practices to integrate such implemented devicesand/or processes and/or systems into more comprehensive devices and/orprocesses and/or systems. That is, at least a portion of the devicesand/or processes and/or systems described herein can be integrated intoother devices and/or processes and/or systems via a reasonable amount ofexperimentation. Those having skill in the art will recognize thatexamples of such other devices and/or processes and/or systems mightinclude—as appropriate to context and application—all or part of devicesand/or processes and/or systems of (a) an air conveyance (e.g., anairplane, rocket, helicopter, etc.), (b) a ground conveyance (e.g., acar, truck, locomotive, tank, armored personnel carrier, etc.), (c) abuilding (e.g., a home, warehouse, office, etc.), (d) an appliance(e.g., a refrigerator, a washing machine, a dryer, etc.), (e) acommunications system (e.g., a networked system, a telephone system, aVoice over IP system, etc.), (f) a business entity (e.g., an InternetService Provider (ISP) entity such as Comcast Cable, Century Link,Southwestern Bell, etc.), or (g) a wired/wireless services entity (e.g.,Sprint, Verizon, AT&T, etc.), etc.

The claims, description, and drawings of this application may describeone or more of the instant technologies in operational/functionallanguage, for example as a set of operations to be performed by acomputer. Such operational/functional description in most instanceswould be understood by one skilled the art as specifically-configuredhardware (e.g., because a general purpose computer in effect becomes aspecial purpose computer once it is programmed to perform particularfunctions pursuant to instructions from program software).

Importantly, although the operational/functional descriptions describedherein are understandable by the human mind, they are not abstract ideasof the operations/functions divorced from computational implementationof those operations/functions. Rather, the operations/functionsrepresent a specification for the massively complex computationalmachines or other means. As discussed in detail below, theoperational/functional language must be read in its proper technologicalcontext, i.e., as concrete specifications for physical implementations.

The logical operations/functions described herein are a distillation ofmachine specifications or other physical mechanisms specified by theoperations/functions such that the otherwise inscrutable machinespecifications may be comprehensible to the human mind. The distillationalso allows one of skill in the art to adapt the operational/functionaldescription of the technology across many different specific vendors'hardware configurations or platforms, without being limited to specificvendors' hardware configurations or platforms.

Some of the present technical description (e.g., detailed description,drawings, claims, etc.) may be set forth in terms of logicaloperations/functions. As described in more detail in the followingparagraphs, these logical operations/functions are not representationsof abstract ideas, but rather representative of static or sequencedspecifications of various hardware elements. Differently stated, unlesscontext dictates otherwise, the logical operations/functions will beunderstood by those of skill in the art to be representative of staticor sequenced specifications of various hardware elements. This is truebecause tools available to one of skill in the art to implementtechnical disclosures set forth in operational/functional formats—toolsin the form of a high-level programming language (e.g., C, java, visualbasic), etc.), or tools in the form of Very high speed HardwareDescription Language (“VHDL,” which is a language that uses text todescribe logic circuits)—are generators of static or sequencedspecifications of various hardware configurations. This fact issometimes obscured by the broad term “software,” but, as shown by thefollowing explanation, those skilled in the art understand that what istermed “software” is a shorthand for a massively complexinterchaining/specification of ordered-matter elements. The term“ordered-matter elements” may refer to physical components ofcomputation, such as assemblies of electronic logic gates, molecularcomputing logic constituents, quantum computing mechanisms, etc.

For example, a high-level programming language is a programming languagewith strong abstraction, e.g., multiple levels of abstraction, from thedetails of the sequential organizations, states, inputs, outputs, etc.,of the machines that a high-level programming language actuallyspecifies. See, e.g., Wikipedia, High-level programming language,http://en.wikipedia.org/wiki/High-level_programming_language (as of Jun.5, 2012, 21:00 GMT) (URL included merely to provide writtendescription). In order to facilitate human comprehension, in manyinstances, high-level programming languages resemble or even sharesymbols with natural languages. See, e.g., Wikipedia, Natural language,http://en.wikipedia.org/wiki/Natural_language (as of Jun. 5, 2012, 21:00GMT) (URL included merely to provide written description).

It has been argued that because high-level programming languages usestrong abstraction (e.g., that they may resemble or share symbols withnatural languages), they are therefore a “purely mental construct.”(e.g., that “software”—a computer program or computer programming—issomehow an ineffable mental construct, because at a high level ofabstraction, it can be conceived and understood in the human mind). Thisargument has been used to characterize technical description in the formof functions/operations as somehow “abstract ideas.” In fact, intechnological arts (e.g., the information and communicationtechnologies) this is not true.

The fact that high-level programming languages use strong abstraction tofacilitate human understanding should not be taken as an indication thatwhat is expressed is an abstract idea. In fact, those skilled in the artunderstand that just the opposite is true. If a high-level programminglanguage is the tool used to implement a technical disclosure in theform of functions/operations, those skilled in the art will recognizethat, far from being abstract, imprecise, “fuzzy,” or “mental” in anysignificant semantic sense, such a tool is instead a nearincomprehensibly precise sequential specification of specificcomputational machines—the parts of which are built up byactivating/selecting such parts from typically more generalcomputational machines over time (e.g., clocked time). This fact issometimes obscured by the superficial similarities between high-levelprogramming languages and natural languages. These superficialsimilarities also may cause a glossing over of the fact that high-levelprogramming language implementations ultimately perform valuable work bycreating/controlling many different computational machines.

The many different computational machines that a high-level programminglanguage specifies are almost unimaginably complex. At base, thehardware used in the computational machines typically consists of sometype of ordered matter (e.g., traditional electronic devices (e.g.,transistors), deoxyribonucleic acid (DNA), quantum devices, mechanicalswitches, optics, fluidics, pneumatics, optical devices (e.g., opticalinterference devices), molecules, etc.) that are arranged to form logicgates. Logic gates are typically physical devices that may beelectrically, mechanically, chemically, or otherwise driven to changephysical state in order to create a physical reality of Boolean logic.

Logic gates may be arranged to form logic circuits, which are typicallyphysical devices that may be electrically, mechanically, chemically, orotherwise driven to create a physical reality of certain logicalfunctions. Types of logic circuits include such devices as multiplexers,registers, arithmetic logic units (ALUs), computer memory, etc., eachtype of which may be combined to form yet other types of physicaldevices, such as a central processing unit (CPU)—the best known of whichis the microprocessor. A modern microprocessor will often contain morethan one hundred million logic gates in its many logic circuits (andoften more than a billion transistors). See, e.g., Wikipedia, Logicgates, http://en.wikipedia.org/wiki/Logic_gates (as of Jun. 5, 2012,21:03 GMT) (URL included merely to provide written description).

The logic circuits forming the microprocessor are arranged to provide amicroarchitecture that will carry out the instructions defined by thatmicroprocessor's defined Instruction Set Architecture. The InstructionSet Architecture is the part of the microprocessor architecture relatedto programming, including the native data types, instructions,registers, addressing modes, memory architecture, interrupt andexception handling, and external Input/Output. See, e.g., Wikipedia,Computer architecture,http://en.wikipedia.org/wiki/Computer_architecture (as of Jun. 5, 2012,21:03 GMT) (URL included merely to provide written description).

The Instruction Set Architecture includes a specification of the machinelanguage that can be used by programmers to use/control themicroprocessor. Since the machine language instructions are such thatthey may be executed directly by the microprocessor, typically theyconsist of strings of binary digits, or bits. For example, a typicalmachine language instruction might be many bits long (e.g., 32, 64, or128 bit strings are currently common). A typical machine languageinstruction might take the form “11110000101011110000111100111111” (a 32bit instruction).

It is significant here that, although the machine language instructionsare written as sequences of binary digits, in actuality those binarydigits specify physical reality. For example, if certain semiconductorsare used to make the operations of Boolean logic a physical reality, theapparently mathematical bits “1” and “0” in a machine languageinstruction actually constitute a shorthand that specifies theapplication of specific voltages to specific wires. For example, in somesemiconductor technologies, the binary number “1” (e.g., logical “1”) ina machine language instruction specifies around +5 volts applied to aspecific “wire” (e.g., metallic traces on a printed circuit board) andthe binary number “0” (e.g., logical “0”) in a machine languageinstruction specifies around −5 volts applied to a specific “wire.” Inaddition to specifying voltages of the machines' configuration, suchmachine language instructions also select out and activate specificgroupings of logic gates from the millions of logic gates of the moregeneral machine. Thus, far from abstract mathematical expressions,machine language instruction programs, even though written as a stringof zeros and ones, specify many, many constructed physical machines orphysical machine states.

Machine language is typically incomprehensible by most humans (e.g., theabove example was just ONE instruction, and some personal computersexecute more than two billion instructions every second). See, e.g.,Wikipedia, Instructions per second,http://en.wikipedia.org/wiki/Instructions_per_second (as of Jun. 5,2012, 21:04 GMT) (URL included merely to provide written description).

Thus, programs written in machine language—which may be tens of millionsof machine language instructions long—are incomprehensible. In view ofthis, early assembly languages were developed that used mnemonic codesto refer to machine language instructions, rather than using the machinelanguage instructions' numeric values directly (e.g., for performing amultiplication operation, programmers coded the abbreviation “mult,”which represents the binary number “011000” in MIPS machine code). Whileassembly languages were initially a great aid to humans controlling themicroprocessors to perform work, in time the complexity of the work thatneeded to be done by the humans outstripped the ability of humans tocontrol the microprocessors using merely assembly languages.

At this point, it was noted that the same tasks needed to be done overand over, and the machine language necessary to do those repetitivetasks was the same. In view of this, compilers were created. A compileris a device that takes a statement that is more comprehensible to ahuman than either machine or assembly language, such as “add 2+2 andoutput the result,” and translates that human understandable statementinto a complicated, tedious, and immense machine language code (e.g.,millions of 32, 64, or 128 bit length strings). Compilers thus translatehigh-level programming language into machine language.

This compiled machine language, as described above, is then used as thetechnical specification which sequentially constructs and causes theinteroperation of many different computational machines such thathumanly useful, tangible, and concrete work is done. For example, asindicated above, such machine language—the compiled version of thehigher-level language—functions as a technical specification whichselects out hardware logic gates, specifies voltage levels, voltagetransition timings, etc., such that the humanly useful work isaccomplished by the hardware.

Thus, a functional/operational technical description, when viewed by oneof skill in the art, is far from an abstract idea. Rather, such afunctional/operational technical description, when understood throughthe tools available in the art such as those just described, is insteadunderstood to be a humanly understandable representation of a hardwarespecification, the complexity and specificity of which far exceeds thecomprehension of most any one human. With this in mind, those skilled inthe art will understand that any such operational/functional technicaldescriptions—in view of the disclosures herein and the knowledge ofthose skilled in the art—may be understood as operations made intophysical reality by (a) one or more interchained physical machines, (b)interchained logic gates configured to create one or more physicalmachine(s) representative of sequential/combinatorial logic(s), (c)interchained ordered matter making up logic gates (e.g., interchainedelectronic devices (e.g., transistors), DNA, quantum devices, mechanicalswitches, optics, fluidics, pneumatics, molecules, etc.) that createphysical reality representative of logic(s), or (d) virtually anycombination of the foregoing. Indeed, any physical object which has astable, measurable, and changeable state may be used to construct amachine based on the above technical description. Charles Babbage, forexample, constructed the first computer out of wood and powered bycranking a handle.

Thus, far from being understood as an abstract idea, those skilled inthe art will recognize a functional/operational technical description asa humanly-understandable representation of one or more almostunimaginably complex and time sequenced hardware instantiations. Thefact that functional/operational technical descriptions might lendthemselves readily to high-level computing languages (or high-levelblock diagrams for that matter) that share some words, structures,phrases, etc. with natural language simply cannot be taken as anindication that such functional/operational technical descriptions areabstract ideas, or mere expressions of abstract ideas. In fact, asoutlined herein, in the technological arts this is simply not true. Whenviewed through the tools available to those of skill in the art, suchfunctional/operational technical descriptions are seen as specifyinghardware configurations of almost unimaginable complexity.

As outlined above, the reason for the use of functional/operationaltechnical descriptions is at least twofold. First, the use offunctional/operational technical descriptions allows near-infinitelycomplex machines and machine operations arising from interchainedhardware elements to be described in a manner that the human mind canprocess (e.g., by mimicking natural language and logical narrativeflow). Second, the use of functional/operational technical descriptionsassists the person of skill in the art in understanding the describedsubject matter by providing a description that is more or lessindependent of any specific vendor's piece(s) of hardware.

The use of functional/operational technical descriptions assists theperson of skill in the art in understanding the described subject mattersince, as is evident from the above discussion, one could easily,although not quickly, transcribe the technical descriptions set forth inthis document as trillions of ones and zeroes, billions of single linesof assembly-level machine code, millions of logic gates, thousands ofgate arrays, or any number of intermediate levels of abstractions.However, if any such low-level technical descriptions were to replacethe present technical description, a person of skill in the art couldencounter undue difficulty in implementing the disclosure, because sucha low-level technical description would likely add complexity without acorresponding benefit (e.g., by describing the subject matter utilizingthe conventions of one or more vendor-specific pieces of hardware).Thus, the use of functional/operational technical descriptions assiststhose of skill in the art by separating the technical descriptions fromthe conventions of any vendor-specific piece of hardware.

In view of the foregoing, the logical operations/functions set forth inthe present technical description are representative of static orsequenced specifications of various ordered-matter elements, in orderthat such specifications may be comprehensible to the human mind andadaptable to create many various hardware configurations. The logicaloperations/functions disclosed herein should be treated as such, andshould not be disparagingly characterized as abstract ideas merelybecause the specifications they represent are presented in a manner thatone of skill in the art can readily understand and apply in a mannerindependent of a specific vendor's hardware implementation.

In certain cases, use of a system or method may occur in a territoryeven if components are located outside the territory. For example, in adistributed computing context, use of a distributed computing system mayoccur in a territory even though parts of the system may be locatedoutside of the territory (e.g., relay, server, processor, signal-bearingmedium, transmitting computer, receiving computer, etc. located outsidethe territory).

A sale of a system or method may likewise occur in a territory even ifcomponents of the system or method are located and/or used outside theterritory.

Further, implementation of at least part of a system for performing amethod in one territory does not preclude use of the system in anotherterritory.

All of the above U.S. patents, U.S. patent application publications,U.S. patent applications, foreign patents, foreign patent applicationsand non-patent publications referred to in this specification and/orlisted in any Application Data Sheet are incorporated herein byreference, to the extent not inconsistent herewith.

The herein described subject matter sometimes illustrates differentcomponents contained within, or connected with, different othercomponents. It is to be understood that such depicted architectures aremerely exemplary, and that in fact many other architectures may beimplemented which achieve the same functionality. In a conceptual sense,any arrangement of components to achieve the same functionality iseffectively “associated” such that the desired functionality isachieved. Hence, any two components herein combined to achieve aparticular functionality can be seen as “associated with” each othersuch that the desired functionality is achieved, irrespective ofarchitectures or intermedial components. Likewise, any two components soassociated can also be viewed as being “operably connected,” or“operably coupled,” to each other to achieve the desired functionality,and any two components capable of being so associated can also be viewedas being “operably couplable,” to each other to achieve the desiredfunctionality. Specific examples of operably couplable include but arenot limited to physically mateable and/or physically interactingcomponents, and/or wirelessly interactable, and/or wirelesslyinteracting components, and/or logically interacting, and/or logicallyinteractable components.

In some instances, one or more components may be referred to herein as“configured to,” “configured by,” “configurable to,” “operable/operativeto,” “adapted/adaptable,” “able to,” “conformable/conformed to,” etc.Those skilled in the art will recognize that such terms (e.g.“configured to”) can generally encompass active-state components and/orinactive-state components and/or standby-state components, unlesscontext requires otherwise.

With respect to the use of substantially any plural and/or singularterms herein, those having skill in the art can translate from theplural to the singular and/or from the singular to the plural as isappropriate to the context and/or application. The varioussingular/plural permutations are not expressly set forth herein for sakeof clarity.

While particular aspects of the present subject matter described hereinhave been shown and described, it will be apparent to those skilled inthe art that, based upon the teachings herein, changes and modificationsmay be made without departing from the subject matter described hereinand its broader aspects and, therefore, the appended claims are toencompass within their scope all such changes and modifications as arewithin the true spirit and scope of the subject matter described herein.It will be understood by those within the art that, in general, termsused herein, and especially in the appended claims (e.g., bodies of theappended claims) are generally intended as “open” terms (e.g., the term“including” should be interpreted as “including but not limited to,” theterm “having” should be interpreted as “having at least,” the term“includes” should be interpreted as “includes but is not limited to,”etc.). It will be further understood by those within the art that if aspecific number of an introduced claim recitation is intended, such anintent will be explicitly recited in the claim, and in the absence ofsuch recitation no such intent is present. For example, as an aid tounderstanding, the following appended claims may contain usage of theintroductory phrases “at least one” and “one or more” to introduce claimrecitations. However, the use of such phrases should not be construed toimply that the introduction of a claim recitation by the indefinitearticles “a” or “an” limits any particular claim containing suchintroduced claim recitation to claims containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (e.g., “a” and/or “an” should typically be interpreted to mean “atleast one” or “one or more”); the same holds true for the use ofdefinite articles used to introduce claim recitations. In addition, evenif a specific number of an introduced claim recitation is explicitlyrecited, those skilled in the art will recognize that such recitationshould typically be interpreted to mean at least the recited number(e.g., the bare recitation of “two recitations,” without othermodifiers, typically means at least two recitations, or two or morerecitations). Furthermore, in those instances where a conventionanalogous to “at least one of A, B, and C, etc.” is used, in generalsuch a construction is intended in the sense one having skill in the artwould understand the convention (e.g., “ a system having at least one ofA, B, and C” would include but not be limited to systems that have Aalone, B alone, C alone, A and B together, A and C together, B and Ctogether, and/or A, B, and C together, etc.). In those instances where aconvention analogous to “at least one of A, B, or C, etc.” is used, ingeneral such a construction is intended in the sense one having skill inthe art would understand the convention (e.g., “ a system having atleast one of A, B, or C” would include but not be limited to systemsthat have A alone, B alone, C alone, A and B together, A and C together,B and C together, and/or A, B, and C together, etc.). It will be furtherunderstood by those within the art that typically a disjunctive wordand/or phrase presenting two or more alternative terms, whether in thedescription, claims, or drawings, should be understood to contemplatethe possibilities of including one of the terms, either of the terms, orboth terms unless context dictates otherwise. For example, the phrase “Aor B” will be typically understood to include the possibilities of “A”or “B” or “A and B.”

With respect to the appended claims, those skilled in the art willappreciate that recited operations therein may generally be performed inany order. Also, although various operational flows are presented in asequence(s), it should be understood that the various operations may beperformed in other orders than those which are illustrated, or may beperformed concurrently. Examples of such alternate orderings may includeoverlapping, interleaved, interrupted, reordered, incremental,preparatory, supplemental, simultaneous, reverse, or other variantorderings, unless context dictates otherwise. Furthermore, terms like“responsive to,” “related to,” or other past-tense adjectives aregenerally not intended to exclude such variants, unless context dictatesotherwise.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopeand spirit being indicated by the following claims.

1-41. (canceled)
 42. An augmented reality device comprising: a display;a camera; a wireless communication interface; circuitry configured foraccepting input associated with a real-world object that is in a fieldof view of the camera; circuitry configured for performing imageprocessing with respect to image data of the real-world object torecognize the real-world object; circuitry configured for accessing viathe wireless communication interface information regarding thereal-world object from one or more remote data sources; and circuitryconfigured for outputting an augmented reality representation includingthe information regarding the real-world object at a position on thedisplay corresponding to the real-world object in the field of view ofthe camera.
 43. The augmented reality device of claim 42, wherein theaugmented reality device comprises a smartphone, a tablet computer, asmartwatch, a digital camera, eyeglasses, or a digital video device. 44.The augmented reality device of claim 42, wherein the circuitryconfigured for accepting input associated with a real-world object thatis in a field of view of the camera comprises: circuitry configured foraccepting at least one of the following types of input associated with areal-world object that is in a field of view of the camera: handgesture, finger gesture, voice input, pointing, touch input, or eyemovement input.
 45. The augmented reality device of claim 42, whereinthe circuitry configured for accepting input associated with areal-world object that is in a field of view of the camera comprises:circuitry configured for accepting input associated with at least one ofthe following types of real-world objects: person, animal, product,sign, building, or vehicle.
 46. The augmented reality device of claim42, wherein the circuitry configured for accepting input associated witha real-world object that is in a field of view of the camera comprises:circuitry configured for accepting input associated with a real-worldobject that is moving in a field of view of the camera.
 47. Theaugmented reality device of claim 42, wherein the circuitry configuredfor accessing via the wireless communication interface informationregarding the real-world object from one or more remote data sourcescomprises: circuitry configured for accessing via the wirelesscommunication interface at least some of the following types ofinformation regarding the real-world object: picture, music, media,social media, or application.
 48. The augmented reality device of claim42, wherein the circuitry configured for accessing via the wirelesscommunication interface information regarding the real-world object fromone or more remote data sources comprises: circuitry configured foraccessing via the wireless communication interface information regardingthe real-world object from one or more of the following remote datasources: payment, music, media, social media, or online.
 49. Theaugmented reality device of claim 42, wherein the circuitry configuredfor outputting an augmented reality representation including theinformation regarding the real-world object at a position on the displaycorresponding to the real-world object in the field of view of thecamera comprises: circuitry configured for outputting a menu foraccessing the information regarding the real-world object at a positionon the display corresponding to the real-world object in the field ofview of the camera.
 50. The augmented reality device of claim 42,wherein the circuitry configured for outputting an augmented realityrepresentation including the information regarding the real-world objectat a position on the display corresponding to the real-world object inthe field of view of the camera comprises: circuitry configured foroutputting a command option for using the information regarding thereal-world object at a position on the display corresponding to thereal-world object in the field of view of the camera.
 51. The augmentedreality device of claim 42, wherein the circuitry configured foroutputting an augmented reality representation including the informationregarding the real-world object at a position on the displaycorresponding to the real-world object in the field of view of thecamera comprises: circuitry configured for tagging the real-world objectwith an augmented reality representation.
 52. The augmented realitydevice of claim 42, wherein the circuitry configured for outputting anaugmented reality representation including the information regarding thereal-world object at a position on the display corresponding to thereal-world object in the field of view of the camera comprises:circuitry configured for outputting a dialog interface including theinformation regarding the real-world object at a position on the displaycorresponding to the real-world object in the field of view of thecamera.
 53. The augmented reality device of claim 42, wherein thecircuitry configured for outputting an augmented reality representationincluding the information regarding the real-world object at a positionon the display corresponding to the real-world object in the field ofview of the camera comprises: circuitry configured for outputting anaugmented reality representation including the information regarding thereal-world object at a position on the display near the real-worldobject in the field of view of the camera.
 54. The augmented realitydevice of claim 42, wherein the circuitry configured for outputting anaugmented reality representation including the information regarding thereal-world object at a position on the display corresponding to thereal-world object in the field of view of the camera comprises:circuitry configured for outputting one or more menu items including theinformation regarding the real-world object based at least partly oncontext or geographical location data.
 55. The augmented reality deviceof claim 42, further comprising: circuitry configured for determining acontext or a geographical location.
 56. The augmented reality device ofclaim 42, further comprising: circuitry configured for detecting anaction based at least partly on a context or a geographical location.57. The augmented reality device of claim 42, further comprising:circuitry configured for generating a digital representation of thereal-world object to preserve intractability with the real-world objectin the field of view of the camera, the digital representation includingone or more of an image, proxy, or simulation.
 58. The augmented realitydevice of claim 42, further comprising: circuitry configured forhighlighting one or more real-world objects within the field of view ofthe camera that are available for selection.
 59. The augmented realitydevice of claim 42, further comprising: circuitry configured for movingthe augmented reality representation in response to an indication ofcompletion.
 60. A system comprising: circuitry configured for acceptinginput associated with a real-world object that is in a field of view ofa camera of an augmented reality device; circuitry configured forperforming image processing with respect to image data of the real-worldobject to recognize the real-world object; circuitry configured foraccessing information regarding the real-world object from one or moreremote data sources; and circuitry configured for outputting anaugmented reality representation including the information regarding thereal-world object at a position corresponding to the real-world objectin the field of view of the camera.
 61. A computer implemented process,the process comprising: accepting input associated with a real-worldobject that is in a field of view of a camera of an augmented realitydevice; performing image processing with respect to image data of thereal-world object to recognize the real-world object; accessinginformation regarding the real-world object from one or more remote datasources; and outputting an augmented reality representation includingthe information regarding the real-world object at a positioncorresponding to the real-world object in the field of view of thecamera.